System and methods of monitoring driver behavior for vehicular fleet management in a fleet of vehicles using driver-facing imaging device

ABSTRACT

Systems and methods monitor driver behavior for vehicular fleet management in a fleet of vehicles using driver-facing imaging device. The systems and methods herein relate generally to vehicular fleet management for enhancing safety of the fleet and improving the performance of the fleet drivers, and further relate to monitoring the operation of fleet vehicles using one or more driver-facing imaging devices disposed in the fleet vehicles for recording activities of the fleet drivers and their passengers, storing information relating to the monitored activities, selectively generating warnings related to the monitored activities, and reporting the monitored activities to a central fleet management system for use in enhancing the safety of the vehicles of the fleet and for helping to improve the performance of the fleet drivers.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/933,624, filed Jul. 20, 2020, entitled: SYSTEM AND METHODS OFMONITORING DRIVER BEHAVIOR FOR VEHICULAR FLEET MANAGEMENT IN A FLEET OFVEHICLES USING DRIVER-FACING IMAGING DEVICE, which is a continuation ofU.S. application Ser. No. 16/741,966, filed Jan. 14, 2020, now U.S. Pat.No. 10,719,725, entitled: SYSTEM AND METHODS OF MONITORING DRIVERBEHAVIOR FOR VEHICULAR FLEET MANAGEMENT IN A FLEET OF VEHICLES USINGDRIVER-FACING IMAGING DEVICE, which is a continuation of U.S.application Ser. No. 15/810,030, filed Nov. 11, 2017, now U.S. Pat. No.10,572,745, entitled: SYSTEM AND METHODS OF MONITORING DRIVER BEHAVIORFOR VEHICULAR FLEET MANAGEMENT IN A FLEET OF VEHICLES USINGDRIVER-FACING IMAGING DEVICE.

This application is related to U.S. application Ser. No. 14/233,319,filed Jul. 12, 2012, now U.S. Pat. No. 9,922,567, entitled: VEHICULARFLEET MANAGEMENT SYSTEM AND METHODS OF MONITORING AND IMPROVING

DRIVER PERFORMANCE IN A FLEET OF VEHICLES, the contents of which isincorporated herein by reference in its entirety.

This application is also related to U.S. application Ser. No.15/810,029, filed Nov. 11, 2017, now U.S. Pat. No. 10,339,401, entitled:SYSTEM AND METHODS OF MONITORING DRIVER BEHAVIOR FOR VEHICULAR FLEETMANAGEMENT IN A FLEET OF VEHICLES USING DRIVER-FACING IMAGING DEVICE,the contents of which is incorporated herein by reference in itsentirety.

This application is also related to U.S. application Ser. No.16/413,913, filed May 16, 2019, now U.S. Pat. No. 10,671,869, entitled:SYSTEM AND METHODS OF MONITORING DRIVER BEHAVIOR FOR VEHICULAR FLEETMANAGEMENT IN A FLEET OF VEHICLES USING DRIVER-FACING IMAGING DEVICE,the contents of which is incorporated herein by reference in itsentirety.

This application is also related to U.S. application Ser. No.16/878,697, filed May 20, 2020, entitled: SYSTEM AND METHODS OFMONITORING DRIVER BEHAVIOR FOR VEHICULAR FLEET MANAGEMENT IN A FLEET OFVEHICLES USING DRIVER-FACING IMAGING DEVICE, the contents of which isincorporated herein by reference in its entirety.

TECHNICAL FIELD

The embodiments herein relate generally to vehicular fleet managementfor enhancing safety of the fleet and improving the performance of thefleet drivers. More specifically, particular embodiments relate tomonitoring the operation of fleet vehicles using one or moredriver-facing imaging devices disposed in the fleet vehicles forrecording activities of the fleet drivers and their passengers, andreporting the monitored activities to a central fleet management systemfor use in enhancing the safety of the vehicles of the fleet and forhelping to improve the performance of the fleet drivers.

BACKGROUND

Existing systems and methods in the vehicular fleet management fieldfocus on specific features of image capture systems and datatransmission of files within the image capture systems. For example,U.S. Pat. No. 7,671,762 to Breslau teaches a system and method oftransceiving vehicle data that involves transmission of data from onevehicle to another. Specifically, Breslau involves transmission andreception of vehicle identification data, and vehicular position data,and includes the use of Global Position Sensor (GPS) signals andsatellite transmission.

Another existing technology is disclosed in U.S. Pat. No. 6,389,340 toRayner wherein a circuit is taught that terminates image capture uponoccurrence of a triggering event, and in which the system components arehoused within a rearview mirror of a vehicle such as a car or truck.

U.S. Pat. No. 7,804,426 to Etcheson teaches a system and method forselective review of event data that comprises computer-assisted cueingof driving data for the selective review in order to save time. Eventdata is continuously captured and sent to a data buffer. The event datais sent to an event detector when requested by a fleet manager or thelike.

In related U.S. application Ser. No. 14/233,319, filed Jul. 12, 2012,entitled: VEHICULAR FLEET MANAGEMENT SYSTEM AND METHODS OF MONITORINGAND IMPROVING DRIVER PERFORMANCE IN A FLEET OF VEHICLES, a system andmethod is described in which vehicles are configured to collect driverand vehicle event data, selectively compress and encode the collecteddriver and vehicle event data, and communicate the compressed andencoded data wirelessly to one or more telematics service providers. Oneor more servers may poll this driver event data periodically, processit, and present multiple methods to end users by which they are able toview and analyze it. The system described permits fleet managers to usethis driver event data, received through a report or notification, orpulled directly from a web-based portal, to monitor, correct and/orreward driver behavior, and to implement driver education and trainingprograms, or the like.

In addition to the above, systems having both forward-facing cameras aswell as driver-facing cameras are known as well. These systems typicallycontinuously capture images of the roadway and of the driver within theinterior of the vehicle, and store the images in a large buffer file,such as a first-in-first out (FOFO) buffer, for example. The roadway anddriver image data is sent to an event detector when requested by a fleetmanager or the like. In that way, the activities of the driver duringany selected event can be determined by “winding back” the video of therecorded vehicle operation to the proper time of the occurrence of theselected event.

It is desirable, however, to more intelligently monitor driver behaviorby monitoring one or more particular behaviors rather than by usinggross imaging and/or by using gross vehicle data collection.

It is further desirable to analyze the one or more particular driverbehaviors, preferably before an occurrence of any significant events, sothat the driver or others such as fleet managers or the like may besuitably warned beforehand, if possible. It is further desirable thatthe drivers may further be graded relative to safety and otherconsiderations, as well as ranked relative to other drivers in the fleetof vehicles, for motivating the drivers to behave better therebyenhancing the overall safety of the fleet and improving overall fleetperformance.

SUMMARY OF THE EXAMPLE EMBODIMENTS

The embodiments herein provide for new and improved systems and methodsof monitoring driver behavior for vehicular fleet management in a fleetof vehicles using a driver-facing imaging device.

In embodiments herein, systems and methods are provided using adriver-facing camera for monitoring driver behavior directly inaccordance with a detected head position of the driver within thevehicle being operated by the driver. Systems and methods are providedusing the driver-facing camera for monitoring the driver's use ofcommercial vehicle mirrors, for monitoring the driver's attention to theroad, for monitoring the driver's head position relative to a properhead position, for monitoring the driver's head pose metric, formonitoring any impediments to the image collected by the driver-facingcamera, and for monitoring the driver's eyes on the road and for makingadjustments on adaptive lane departure warning system of the associatedvehicle. These driver behaviors may be directly monitored as well asothers as may be necessary and/or desired in accordance with theembodiments herein.

In further embodiments herein, systems and methods are provided using adriver-facing camera for monitoring driver behavior indirectly inaccordance with detected aspects of components of the interior of thevehicle being operated by the driver. Systems and methods are providedusing the driver-facing camera for monitoring the driver's proper use ofthe vehicle seatbelt, for monitoring the driver's proper hand positionson the steering when, and for monitoring the driver's compliance withfleet policies relative to unauthorized passengers being in the vehicle.These driver behaviors may be directly monitored as well as others asmay be necessary and/or desired in accordance with the embodimentsherein.

In accordance with embodiments herein, systems, methods and logic areprovided including various vehicle sensors and a driver facing camerafor determining when a set of one or more predetermined conditions of avehicle are met or otherwise satisfied, determining a driver's headpose, learning or otherwise training the system on average values of thedriver head pose (pitch, yaw, roll, etc.) when the set of one or morepredetermined conditions of the vehicle are met or otherwise satisfied,and determining any occurrences of driver head pose deviations from theaverage values.

In accordance with embodiments herein, systems, methods and logic areprovided including various vehicle sensors and a driver facing camerafor determining a driver's head pose, learning or otherwise training thesystem on a head pose distribution and/or a head pose heat map, anddetermining any occurrences of driver head pose deviations from the ahead pose distribution and/or a head pose heat map average values.

In accordance with embodiments herein, systems, methods and logic areprovided including various vehicle sensors for determining when a set ofone or more predetermined conditions propitious for determininginfractions or driver misbehavior are met or otherwise satisfied such asfor example a vehicle door status, a speed change, an unusual stoppinglocation, unauthorized passenger visible, or the like, and a driverfacing camera for obtaining images of the cabin of the vehicle inresponse to the set of one or more predetermined conditions of thevehicle are met.

In accordance with embodiments herein, systems, methods and logic areprovided including various vehicle sensors and a driver facing camerafor learning or otherwise training the system on average values ofappearance (template images or descriptions) of vehicle cabin items,such as seat belt buckles, empty seats, steering wheel, door edges,mirror locations, and determining any occurrences of changes ordeviations from the average or learned operational set values of thelearned template images or descriptions.

In accordance with embodiments herein, systems, methods and logic areprovided including various vehicle sensors and a driver facing camerafor determining a driver's head pose vector, learning or otherwisetraining the system on average values of the driver's head pose vector,and selectively adapting other system values as a function of thedriver's head pose vector when a persistent deviation from the driverlooking at the road or the driver looking at the mirrors occurs.

In accordance with embodiments herein, systems, methods and logicprovide multi-factor authentication using multiple sensors and a driverfacing camera for driver identity verification using driver image datain combination with and voice print data of the driver, such as forexample by imaging the driver using the driver facing camera, verifyinga visual identity of the driver in accordance with driver databaseinformation and the driver image data obtaining voiceprint data of thedriver uttering a standardized pass phrase while in the field of thedriver facing camera verifying voiceprint identity of the driverrequiring the driver to speak his name, leading to a standardizedcomparison template, and recording the protocol into a local memory ofthe system in the vehicle.

The term “processor means” as used herein refers to any microprocessor,discrete logic (e.g., ASIC), analog circuit, digital circuit, programmedlogic device, memory device containing instructions, and so on. The term“processor means” also refers to “logic” which may include one or moregates, combinations of gates, other circuit components, hardware,firmware, software in execution on a machine, and/or combinations ofeach to perform a function(s) or an action(s), and/or to cause afunction or action from another logic, method, and/or system, a softwarecontrolled microprocessor, a discrete logic (e.g., ASIC), an analogcircuit, a digital circuit, a programmed logic device, a memory devicecontaining instructions, and so on. The term “memory means” as usedherein refers to any non-transitory media that participates in storingdata and/or in providing instructions to the processor means forexecution. Such a non-transitory medium may take many forms, includingbut not limited to volatile and non-volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks. Volatile mediaincludes dynamic memory for example and does not include transitorysignals, carrier waves, or the like. Common forms of computer-readablemedia include, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, papertape, any other physical medium with patternsof holes, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, or any other tangible non-transitory medium from which acomputer can read.

Other embodiments, features and advantages of the example embodimentswill become apparent from the following description of the embodiments,taken together with the accompanying drawings, which illustrate, by wayof example, the principles of the example embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings which are incorporated in and constitute apart of the specification, embodiments of the invention are illustrated,which, together with a general description of the invention given above,and the detailed description given below, serve to exemplify theembodiments of this invention.

FIG. 1 is a diagram of an overview of the fleet management system anduser layout according to the example embodiment.

FIG. 2 depicts operation of an exemplary fleet vehicle operating in aplatoon and having a driver behavior monitoring system having a driverfacing camera in accordance with an embodiment.

FIG. 3 is a schematic illustration of an exemplary embodiment of a datacollection module portion of a driver behavior monitoring system havinga driver facing camera according to the example embodiment;

FIG. 4 is a block diagram that illustrates a computer system suitablefor monitoring driver behavior directly in accordance with a detectedhead position of the driver within the vehicle being operated by thedriver and for monitoring driver behavior indirectly in accordance withdetected aspects of components of the interior of the vehicle beingoperated by the driver in accordance with an example embodiment.

FIG. 4a is a block diagram that illustrates executable logic componentsof the driver behavior monitoring system having a driver facing cameraaccording to the example embodiment.

FIG. 5a is a schematic diagram showing a driver facing imager inaccordance with an example embodiment disposed in the cab of anassociated vehicle in a fixed location at the upper top of a windshieldof the associated vehicle.

FIG. 5b is a diagram of an embodiment of the driver facing imager ofFIG. 5a formed as a driver facing camera in accordance with an exampleembodiment.

FIG. 6a is a first example of a calibration image generated by thedriver facing camera of FIG. 5b and obtained during a first calibrationoperation of the driver behavior monitoring system.

FIG. 6b is an example of a second calibration image generated by thedriver facing camera of FIG. 5b and obtained during a second calibrationoperation of the driver behavior monitoring system.

FIG. 7 is an example of an image generated by the driver facing cameraof FIG. 5b and obtained by the driver behavior monitoring system duringoperation of the associated vehicle.

FIG. 8 is a flow diagram showing a method of operating a driver behaviormonitoring system having a driver facing camera for implementing adriver behavior monitoring and reporting strategy in accordance with anexample embodiment.

FIG. 9 is a flow diagram showing a method of operating a driver behaviormonitoring system having a driver facing camera for implementing apassenger detecting, counting, monitoring, and reporting strategy inaccordance with an example embodiment.

FIG. 9a is a flow diagram showing a further method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a passenger detecting, counting, monitoring, and reportingstrategy in accordance with an example embodiment.

FIG. 10 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a seatbelt usage detection, monitoring, and reportingstrategy in accordance with an example embodiment.

FIG. 10a is a flow diagram showing details of a portion of the method ofoperating a driver behavior monitoring system having a driver facingcamera for implementing the seatbelt usage detection, monitoring, andreporting strategy of FIG. 10, in accordance with an example embodiment.

FIG. 10b is a flow diagram showing further details of a portion of themethod of operating a driver behavior monitoring system having a driverfacing camera for implementing the seatbelt usage detection, monitoring,and reporting strategy of FIG. 10, in accordance with an exampleembodiment.

FIG. 11 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a hands on the steering wheel detection, monitoring, andreporting strategy in accordance with an example embodiment.

FIG. 12 is an example of an image generated by the driver facing cameraof FIG. 5b and obtained by the driver behavior monitoring system duringoperation of the associated vehicle and showing a typical driver havinghis hands on the steering wheel.

FIG. 13 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a driver road attention detection, monitoring, andreporting strategy in accordance with an example embodiment.

FIG. 14 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing an impeded view detection, monitoring, and reportingstrategy in accordance with an example embodiment.

FIG. 15 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a driver's head is out of position detection, monitoring,and reporting strategy in accordance with an example embodiment.

FIG. 15a is a flow diagram showing a further method of operating adriver behavior monitoring system having a driver facing camera forimplementing a driver's head is out of position detection, monitoring,and reporting strategy in accordance with an example embodiment.

FIG. 16 is a schematic diagram showing characteristics of a driver'shead for purposes of determining a driver's head pose vector inaccordance with an example embodiment.

FIG. 17 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera for detecting,monitoring, and reporting whether the driver's head pose distribution issignificantly changing or unacceptable implementing a driver roadattention strategy in accordance with an example embodiment.

FIG. 18 is an example of a head pose distribution map in accordance withan example embodiment.

FIG. 19 is a flow diagram showing a method of comparing driver head posehistograms, and determining and reporting deviations and/or changesbetween the driver head pose histograms.

FIG. 19a is a flow diagram showing a method of comparing head posestatistics, and determining and reporting deviations between a driver'shead pose and desired, situation appropriate, statistics in accordancewith an example embodiment.

FIG. 20 is a flow diagram showing a method of comparing head posehistograms, and determining and reporting deviations between a driver'shead pose and desired, situation appropriate, histograms in accordancewith an example embodiment.

FIG. 21 is an illustration of the bounds applying to mirror usage inaccordance with an example embodiment.

DETAILED DESCRIPTION OF THE EXAMPLE EMBODIMENTS

In the following description of the present invention reference is madeto the accompanying figures which form a part thereof, and in which isshown, by way of illustration, exemplary embodiments illustrating theprinciples of the present invention and how it is practiced. Otherembodiments can be utilized to practice the present invention andstructural and functional changes can be made thereto without departingfrom the scope of the present invention.

Referring now to the drawings, wherein the showings are for the purposeof illustrating the example embodiments for monitoring driver behaviordirectly using a driver-facing camera in accordance with a detected headposition of the driver within the vehicle being operated by the vehicle,and for monitoring driver behavior indirectly using a driver-facingcamera in accordance with detected aspects of components of the interiorof the vehicle being operated by the driver only, and not for purposesof limiting the same, FIG. 1 illustrates an overview of a fleetmanagement and reporting system 100 in accordance with the exampleembodiment. In the example embodiment of the present invention, vehicles110, such as trucks and cars, and particularly fleet vehicles 112, areconfigured with one or more data collection and reporting devices 200(FIG. 2) that generate event data such as, in the example of a fleet oftrucks, truck start, truck stop, and safety event data, wherein one suchsystem includes for example a Lane Departure Warning (LDW) system 322(FIG. 3) that generates signals indicative of one or more events anddriver and vehicle event data regarding in the example of the fleet oftrucks, truck lane wandering or crossing. Additionally, secondarysystems to be described in greater detail below with reference to FIG. 3carried by the vehicles or installed in the vehicle systems such as oneor more video cameras, radar, transmission, engine, tire pressuremonitoring and braking systems for example may generate additionalsafety event data. Third-party systems that generate proprietary safetyevents or data representative of detected safety events may also beinvolved. For example, the embodiments of the present invention mayinclude software code implementing a Bendix® Wingman® ACB systemavailable from Bendix Commercial Vehicle Systems LLC that capturesproprietary safety events and other data relating to the proprietarysafety events and/or relating to the operation of the vehicle by one ormore vehicle operators or drivers.

With continued reference to FIG. 1, these events and event data 120 are,in the example embodiment, selectively sent via one or more wirelessnetworks or wireless links 122 to network servers 132 of one or moreservice providers 130. Wireless service providers 130 utilize servers132 (only one shown for ease of illustration) that collect the wirelessdata 120 provided by the trucks 112. Each also provides a web service bywhich users can report on or download data.

One or more servers 140 of the fleet management and reporting system 100are configured to selectively download or otherwise retrieve data fromthe collection servers 132 which may be third party servers from one ormore various telematics suppliers such as for example those availablefrom PeopleNet Communications Corp. or Qualcomm Inc. for example. Theone or more servers 140 of the fleet management and reporting system 100are configured to initiate processing of the vehicular events andvehicular event data in manners to be described in greater detail below.A web application 142 executable on the one or more servers 140 of thefleet management and reporting system 100 includes a dynamic graphicaluser interface for fleet managers 160 and administrators 162 to view allof the information once it is processed. The subject fleet managementand reporting system 100 of the example embodiment also includes one ormore databases 150 configured to selectively store all event informationprovided from the vehicles 112 in the fleet 110 for one or moredesignated time intervals, including raw and post-processed trip data.

In accordance with the example embodiment, the system administrators 162are users who are provided with interfaces to configure and managefleets, monitor platform performance, view alerts issued by theplatform, and view raw event data and subsequent processing logs and/orviews. Fleet managers 160 may view event information for theirrespective fleet for internal processing. These events can arrive viauser-initiated reports 170 in the web application 142 executable on theone or more servers 140, or via email or other notifications 172. Fleetmanagers 160 may, depending on internal policies and processes or forother reasons, also interface with individual drivers 164 regardingperformance goals, corrections, reports, or coaching.

The subject fleet management and reporting system 100 of the exampleembodiment therefore offers a long list of functions and features to theend user. All have been designed to be driver centric, so that fleetmanagers 160 may focus their attention on driver education, training,and performance improvement. One of the primary beneficial and noveluses of the system 100 is the ease of access to driverspecific-performance data and the ability to normalize each driver'sperformance to compare with the drivers of the fleet as a whole in orderto pinpoint exemplary drivers for commendation as well as those in needof coaching or other corrective action.

FIG. 2 depicts operation of an exemplary fleet vehicle operating in abasic platoon A including a host or leader vehicle 10 in traffic with asecond or follower vehicle 20 in accordance with the present disclosure.As shown, the follower vehicle 20 is traveling proximate to the leadervehicle 10 in an ordered platoon A along a roadway 1. The followervehicle 20 is provided with an electronic control system 12′ whichincludes a data collection and communication module portion 300′ and amonitoring control portion 400′ to be described in greater detail below.Similarly, the leader vehicle 10 is also provided with an equivalentelectronic control system 12 which includes an equivalent datacollection and communication module portion 300 and an equivalentmonitoring control portion 400. In the example embodiments to bedescribed herein, although each of the two or more vehicles comprisingthe various platoons that will be described include the same orequivalent electronic control system 12, 12′ the same or equivalent datacollection and communication module portion 300, 300′ and the same orequivalent monitoring control portion 400,400′ other disparate controlsystems having the functionality to be described herein may equivalentlybe used as necessary or desired.

In the example embodiment illustrated, the electronic control systems12, 12′ of the respective vehicles 20, 10 are configured for mutuallycommunicating signals and exchanging data between each other, and alsofor communicating signals and exchanging data with various othercommunication systems including for example a remote wirelesscommunication system 250 and a remote satellite system 260. These remotesystems 250, 260 can provide, for example, global position system (GPS)data to the vehicles 10, 20 as desired. Other information may beprovided or exchanged between the vehicles and the remote systems aswell such as, for example, fleet management and control data from aremote fleet management facility, or the like (not shown). Although thisfunctionality is provided, the embodiments herein find this remotecommunication, though useful, not necessarily essential wherein theembodiments herein are directed to monitoring driver behavior directlyin accordance with a detected head position of the driver within thevehicle being operated by the driver and for monitoring driver behaviorindirectly in accordance with detected aspects of components of theinterior of the vehicle being operated by the driver without the need toconsult with or act under the direction of or in concert with the remotewireless communication system 250, the remote satellite system 260, theremote fleet management facility, Central Command Center (CCC), aNetwork Operations Center (NOC), or the like.

In addition to the above, the electronic control systems 12, 12′ of eachvehicle 10, 20 operates to perform various vehicle-to-(single)vehicle(V2V Unicast) communication (communication between a broadcastingvehicle and a single responding vehicle), as well as variousvehicle-to-(multiple)vehicle (V2V Broadcast) communication(communication between a broadcasting vehicle and two or more respondingvehicles), and further as well as various vehicle-to-infrastructure(V2I) communication. Preferably, the local V2V Unicast and V2V Broadcastcommunication follows the J2945 DSRC communications specification. Inthis regard, the vehicles forming the basic platoon A can communicatewith each other locally for self-ordering and spacing into a platoonwithout the need for input from the CCC in accordance with theembodiments herein. The vehicles forming the basic platoon A can alsocommunicate with one or more other vehicles locally without the need forinput from the CCC for negotiating the one or more other vehicles intothe platoon in accordance with the embodiments herein. The vehiclesforming the basic platoon A can further communicate with a fleetmanagement facility remotely as may be necessary and/or desired formonitoring driver behavior directly in accordance with a detected headposition of the driver within the vehicle being operated by the driverand for monitoring driver behavior indirectly in accordance withdetected aspects of components of the interior of the vehicle beingoperated by the driver in accordance with further example embodimentsherein.

As noted above, preferably, the local V2V Unicast and V2V Broadcastcommunication between vehicles as will be described herein follows theJ2945 DSRC communications specification. This specification at present,does not define one-to-one vehicle communications. Rather,operationally, each communication-capable vehicle sends the neededinformation by a broadcast to every other communication-capable vehiclewithin range, and the receiving vehicle(s) decide if they want toprocess the received message. For example only vehicles who are Platooncapable and for which the driver has indicated, via a switch or userinterface, that joining a platoon is desired, that vehicle will startbroadcasting and listening for the Platoon protocol messages. All othervehicles in the area may ignore the platoon information. Accordingly, aswill be used herein and for purposes of describing the exampleembodiments, “V2V Unicast” communication will refer to communicationbetween a broadcasting vehicle and a single responding vehicle, and “V2VBroadcast communication” will refer to communication between abroadcasting vehicle and two or more responding vehicles. It is to beappreciated that “V2V Unicast” communication also refers to one-to-onedirect vehicle communications as the J2945 DSRC communicationsspecification is further developed or by use of any one or more otherstandards, specifications, or technologies now known or hereinafterdeveloped.

FIG. 3 is a schematic block diagram depiction that illustrates detailsof the towing vehicle data collection and communication module portion300 of FIG. 2 in accordance with an example embodiment. According toprinciples of the example embodiment as illustrated, the towing vehicledata collection and communication module portion 300 may be adapted todetect, monitor, and report a variety of operational parameters andconditions of the commercial vehicle and the driver's interactiontherewith, and to selectively intervene and take corrective action asmay be needed or desired such as, for example, to maintain vehiclestability or to maintain the vehicle following distance relative toother vehicles within a platoon. In the exemplary embodiment of FIG. 3,the data collection and communication module portion 300 may include oneor more devices or systems 314 for providing input data indicative ofone or more operating parameters or one or more conditions of acommercial vehicle. For example, the devices 314 may be one or moresensors, such as but not limited to, one or more wheel speed sensors316, one or more acceleration sensors such as multi-axis accelerationsensors 317, a steering angle sensor 318, a brake pressure sensor 319,one or more vehicle load sensors 320, a yaw rate sensor 321, a lanedeparture warning (LDW) sensor or system 322, one or more engine speedor condition sensors 323, and a tire pressure (TPMS) monitoring system324. The towing vehicle data collection and communication module portion300 may also utilize additional devices or sensors in the exemplaryembodiment including for example a forward distance sensor 360, and arear distance sensor 362. Other sensors and/or actuators or powergeneration devices or combinations thereof may be used of otherwiseprovided as well, and one or more devices or sensors may be combinedinto a single unit as may be necessary and/or desired.

The towing vehicle data collection and communication module portion 300may also include a logic applying arrangement such as a controller orprocessor 330 and control logic 331, in communication with the one ormore devices or systems 314. The processor 330 may include one or moreinputs for receiving input data from the devices or systems 314. Theprocessor 330 may be adapted to process the input data and compare theraw or processed input data to one or more stored threshold values, orto process the input data and compare the raw or processed input data toone or more circumstance-dependent desired value. The processor 330 mayalso include one or more outputs for delivering a control signal to oneor more vehicle systems 323 based on the comparison. The control signalmay instruct the systems 323 to intervene in the operation of thevehicle to initiate corrective action, and then report this correctiveaction to a wireless service (not shown) or simply store the datalocally to be used for determining a driver quality. For example, theprocessor 330 may generate and send the control signal to an engineelectronic control unit or an actuating device to reduce the enginethrottle 334 and slowing the vehicle down. Further, the processor 330may send the control signal to one or more vehicle brake systems 335,336 to selectively engage the brakes. In the tractor-trailer arrangementof the example embodiment, the processor 330 may engage the brakes 336on one or more wheels of a trailer portion of the vehicle via a trailerpressure control device (not shown), and the brakes 335 on one or morewheels of a tractor portion of the vehicle 12, and then report thiscorrective action to the wireless service or simply store the datalocally to be used for determining a driver quality. A variety ofcorrective actions may be possible and multiple corrective actions maybe initiated at the same time.

The controller 300 may also include a memory portion 340 for storing andaccessing system information, such as for example the system controllogic 331 and control tuning. The memory portion 340, however, may beseparate from the processor 330. The sensors 314 and processor 330 maybe part of a preexisting system or use components of a preexistingsystem. For example, the Bendix® ABS-6™ Advanced Antilock BrakeController with ESP® Stability System available from Bendix CommercialVehicle Systems LLC may be installed on the vehicle. The Bendix® ESP®system may utilize some or all of the sensors described in FIG. 3. Thelogic component of the Bendix® ESP® system resides on the vehicle'santilock brake system electronic control unit, which may be used for theprocessor 330 of the present invention. Therefore, many of thecomponents to support the towing vehicle controller 330 of the presentinvention may be present in a vehicle equipped with the Bendix® ESP®system, thus, not requiring the installation of additional components.The towing vehicle controller 330, however, may utilize independentlyinstalled components if desired. Further, an IMX,6 processor separatefrom the ESP system may execute the functions described herein.

The data collection and communication module portion 300 of the towingvehicle controller 12 may also include a source of input data 342indicative of a configuration/condition of a commercial vehicle. Theprocessor 330 may sense or estimate the configuration/condition of thevehicle based on the input data, and may select a control tuning mode orsensitivity based on the vehicle configuration/condition. The processor330 may compare the operational data received from the sensors orsystems 314 to the information provided by the tuning. The tuning of thesystem may include, but is not be limited to: the nominal center ofgravity height of the vehicle, look-up maps and/or tables for lateralacceleration level for rollover intervention, look-up maps and/or tablesfor yaw rate differential from expected yaw rate for yaw controlinterventions, steering wheel angle allowance, tire variation allowance,and brake pressure rates, magnitudes and maximums to be applied duringcorrective action.

A vehicle configuration/condition may refer to a set of characteristicsof the vehicle which may influence the vehicle's stability (roll and/oryaw). For example, in a vehicle with a towed portion, the source ofinput data 342 may communicate the type of towed portion. Intractor-trailer arrangements, the type of trailer being towed by thetractor may influence the vehicle stability. This is evident, forexample, when multiple trailer combinations (doubles and triples) aretowed. Vehicles with multiple trailer combinations may exhibit anexaggerated response of the rearward units when maneuvering (i.e.rearward amplification). To compensate for rearward amplification, thetowing vehicle controller 330 may select a tuning that makes the systemmore sensitive (i.e. intervene earlier than would occur for a singletrailer condition). The control tuning may be, for example, specificallydefined to optimize the performance of the data collection andcommunication module for a particular type of trailer being hauled by aparticular type of tractor. Thus, the control tuning may be differentfor the same tractor hauling a single trailer, a double trailercombination, or a triple trailer combination.

The type of load the commercial vehicle is carrying and the location ofthe center of gravity of the load may also influence vehicle stability.For example, moving loads such as liquid tankers with partially filledcompartments and livestock may potentially affect the turning androllover performance of the vehicle. Thus, a more sensitive controltuning mode may be selected to account for a moving load. Furthermore, aseparate control tuning mode may be selectable when the vehicle istransferring a load whose center of gravity is particularly low orparticularly high, such as for example with certain types of bigmachinery or low flat steel bars.

In addition, the controller 300 is operatively coupled with one or moredriver facing imaging devices shown in the example embodiment forsimplicity and ease of illustration as a single driver facing camera 345representation of one or more physical video cameras disposed on thevehicle such as, for example, a video camera on each corner of thevehicle, one or more cameras mounted remotely and in operativecommunication with the controller 330 such as a forward facing camera(FFC) disposed on the vehicle in a manner to record images of theroadway ahead of the vehicle, or, as in the example embodiment, in thecab of a commercial vehicle trained on the driver and/or trained on theinterior of the cab of the commercial vehicle. In the exampleembodiments, driver behavior is monitored directly using the driverfacing camera 345 in accordance with a detected head position of thedriver within the vehicle being operated by the vehicle, the details ofwhich will be elaborated below. In further example embodiments, thedriver behavior is monitored directly using the driver facing camera 345in accordance with a detected head pose of the driver. For purposes ofthis description of the example embodiments and for ease of reference,“head pose” is that set of angles describing the orientation of thedriver's head, that is, pitch (driver looking down or up), yaw (driverlooking left or right), and roll (driver tilting his/her head to theleft or right). In still further embodiments, driver behavior ismonitored indirectly using the driver facing camera 345 in accordancewith detected aspects of components of the interior of the vehicle beingoperated by the vehicle, the details of which will be elaborated below.The driver facing camera 345 may include an imager available fromOminivision™ as part/model number 10635, although any other suitableequivalent imager may be used as necessary or desired.

Still yet further, the controller 300 may also include atransmitter/receiver (transceiver) module 350 such as, for example, aradio frequency (RF) transmitter including one or more antennas 352 forwireless communication of the automated deceleration requests, GPS data,one or more various vehicle configuration and/or condition data, or thelike between the vehicles and one or more destinations such as, forexample, to one or more wireless services (not shown) having acorresponding receiver and antenna. The transmitter/receiver(transceiver) module 350 may include various functional parts of subportions operatively coupled with the platoon control unit including forexample a communication receiver portion, a global position sensor (GPS)receiver portion, and a communication transmitter. For communication ofspecific information and/or data, the communication receiver andtransmitter portions may include one or more functional and/oroperational communication interface portions as well.

The processor 330 is operative to communicate the acquired data to theone or more receivers in a raw data form, that is without processing thedata, in a processed form such as in a compressed form, in an encryptedform or both as may be necessary or desired. In this regard, theprocessor 330 may combine selected ones of the vehicle parameter datavalues into processed data representative of higher level vehiclecondition data such as, for example, data from the multi-axisacceleration sensors 317 may be combined with the data from the steeringangle sensor 318 to determine excessive curve speed event data. Otherhybrid event data relatable to the vehicle and driver of the vehicle andobtainable from combining one or more selected raw data items form thesensors includes, for example and without limitation, excessive brakingevent data, excessive curve speed event data, lane departure warningevent data, excessive lane departure event data, lane change withoutturn signal event data, loss of video tracking event data, LDW systemdisabled event data, distance alert event data, forward collisionwarning event data, haptic warning event data, collision mitigationbraking event data, ATC event data, ESC event data, RSC event data, ABSevent data, TPMS event data, engine system event data, average followingdistance event data, average fuel consumption event data, and averageACC usage event data. Importantly, however, and in accordance with theexample embodiments described herein, the controller 300 is operative tostore the acquired image data of the driver and/or of the interior ofthe vehicle in the memory 340, and to selectively communicate theacquired driver and vehicle interior image data to the one or morereceivers via the transceiver 350.

In the example embodiment illustrated, the towing vehicle controllers12, 12′ (FIG. 2) of the respective vehicles of the platoon areconfigured for mutually communicating signals and exchanging databetween each other and between their respective one or more towedvehicles, and also for communicating signals and exchanging data withvarious other communication systems including for example a remotewireless communication system and a remote satellite system. Theseremote systems can provide, for example, global position system (GPS)data to the vehicles as desired. Other information may be provided orexchanged between the vehicles and the remote systems as well such as,for example, fleet management and control data may be received from aremote fleet management facility, or the like (not shown), and driverbehavior data may be sent to the remote fleet management facility, aremote satellite system, a Network Operations Center (NOC), a CentralCommand Center (CCC), or the like.

The towing vehicle controller 300 of FIG. 3 is suitable for executingembodiments of one or more software systems or modules that performtrailer brake strategies and trailer braking control methods accordingto the subject application. The example towing vehicle controller 22 mayinclude a bus or other communication mechanism for communicatinginformation, and a processor 330 coupled with the bus for processinginformation. The computer system includes a main memory 340, such asrandom access memory (RAM) or other dynamic storage device for storinginformation and instructions to be executed by the processor 330, andread only memory (ROM) or other static storage device for storing staticinformation and instructions for the processor 330. Other storagedevices may also suitably be provided for storing information andinstructions as necessary or desired.

Instructions may be read into the main memory 340 from anothercomputer-readable medium, such as another storage device of via thetransceiver 350. Execution of the sequences of instructions contained inmain memory 340 causes the processor 330 to perform the process stepsdescribed herein. In an alternative implementation, hard-wired circuitrymay be used in place of or in combination with software instructions toimplement the invention. Thus implementations of the example embodimentsare not limited to any specific combination of hardware circuitry andsoftware.

In accordance with the descriptions herein, the term “computer-readablemedium” as used herein refers to any non-transitory media thatparticipates in providing instructions to the processor 330 forexecution. Such a non-transitory medium may take many forms, includingbut not limited to volatile and non-volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks. Volatile mediaincludes dynamic memory for example and does not include transitorysignals, carrier waves, or the like. Common forms of computer-readablemedia include, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, papertape, any other physical medium with patternsof holes, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, or any other tangible non-transitory medium from which acomputer can read.

In addition and further in accordance with the descriptions herein, theterm “logic”, as used herein with respect to the Figures, includeshardware, firmware, software in execution on a machine, and/orcombinations of each to perform a function(s) or an action(s), and/or tocause a function or action from another logic, method, and/or system.Logic may include a software controlled microprocessor, a discrete logic(e.g., ASIC), an analog circuit, a digital circuit, a programmed logicdevice, a memory device containing instructions, and so on. Logic mayinclude one or more gates, combinations of gates, or other circuitcomponents.

FIG. 4 is a block diagram that illustrates a driver behavior monitoringcomputer system 400 suitable for executing embodiments of one or moresoftware systems or modules that perform the driver behavior monitoringand reporting analyses according to the subject application. The examplesystem includes a bus 402 or other communication mechanism forcommunicating information, and a processor 404 coupled with the bus forprocessing information. The computer system 400 includes a main memory,such as random access memory (RAM) 406 or other dynamic storage devicefor storing information and instructions to be executed by the processor404, and read only memory (ROM) 408 or other static storage device forstoring static information and instructions for the processor 404. Alogic storage device 410 is also suitably provided for storinginstructions for execution by the processor, and other informationincluding for example one or more calibration values of directlymonitored parameters of the driver, such as proper driver head position,for example, and/or one or more calibration values of indirectlymonitored parameters of the driver, such as proper seat belt usage, forexample. In addition, operator interfaces are provided in the form of aninput device 414 such as a keyboard or a voice recognition inputincluding a microphone and logic transforming human voice sounds intocomputer commands, a human readable display 412 for presenting visibleinformation to the driver, and a cursor control 416 such as a joystickor mouse or the like.

The example embodiments described herein are related to the use of thecomputer system 400 for accessing, aggregating, manipulating anddisplaying information from one or more resources such as, for example,from the driver facing camera 345.

According to one implementation, information from the driver facingcamera 345 is provided by computer system 400 in response to theprocessor 404 executing one or more sequences of one or moreinstructions contained in main memory 406. Such instructions may be readinto main memory 406 from another computer-readable medium, such aslogic storage device 410. The logic storage device 410 may store one ormore subsystems or modules to perform the direct driver behaviormonitoring as set forth herein and/or one or more subsystems or modulesto perform the indirect driver behavior monitoring as set forth herein.Execution of the sequences of instructions contained in main memory 406causes the processor 404 to perform the process steps described herein.In an alternative implementation, hard-wired circuitry may be used inplace of or in combination with software instructions to implement theinvention. Thus implementations of the example embodiments are notlimited to any specific combination of hardware circuitry and software.

In accordance with the descriptions herein, the term “computer readablemedium” as used herein refers to any non-transitory media thatparticipates in providing instructions to the processor 404 forexecution. Such a non-transitory medium may take many forms, includingbut not limited to volatile and non-volatile media. Nonvolatile mediaincludes, for example, optical or magnetic disks. Volatile mediaincludes dynamic memory for example and does not include transitorysignals, carrier waves, or the like. Common forms of computer-readablemedia include, for example, a floppy disk, a flexible disk, hard disk,magnetic tape, or any other magnetic medium, a CDROM, any other opticalmedium, punch cards, papertape, any other physical medium with patternsof holes, a RAM, PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, or any other tangible non-transitory medium from which acomputer can read.

In addition and further in accordance with the descriptions herein, theterm “logic”, as used herein with respect to the Figures, includeshardware, firmware, software in execution on a machine, and/orcombinations of each to perform a function(s) or an action(s), and/or tocause a function or action from another logic, method, and/or system.Logic may include a software controlled microprocessor, a discrete logic(e.g., ASIC), an analog circuit, a digital circuit, a programmed logicdevice, a memory device containing instructions, and so on. Logic mayinclude one or more gates, combinations of gates, or other circuitcomponents.

The driver behavior monitoring computer system 400 includes acommunication interface 418 coupled to the bus 402 which provides atwo-way data communication coupling to a network link 420 that isconnected to local network 422. For example, communication interface 418may be an integrated services digital network (ISDN) card or a modem toprovide a data communication connection to a corresponding type oftelephone line. As another example, communication interface 418 may be alocal area network (LAN) card to provide a data communication connectionto a compatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 418 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424supporting a database 425 storing internal proprietary data and/or todata equipment operated by an Internet Service Provider (ISP) 426. ISP426 in turn provides data communication services through the Internet428. Local network 422 and Internet 428 both use electric,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from the driver behavior monitoring computer system 400, areexemplary forms of carrier waves transporting the information.

The driver behavior monitoring computer system 400 can send messages andreceive data, including program code, through the network(s), networklink 420 and communication interface 418. In the Internet-connectedexample embodiment, the driver behavior monitoring computer system 400is operatively connected with a plurality of external public, private,governmental or commercial servers (not shown) as one or more wirelessservices (not shown) configured to execute a web application inaccordance with the example embodiment to be described below in greaterdetail. In the example embodiment shown, the first server 430 is coupledwith a database 450 storing selected data received by a first wirelessservice such as for example data from a first telematics supplier, thesecond first server 432 is coupled with a database 452 storing selecteddata received by a second wireless service such as for example data froma second telematics supplier, and the third server 434 is coupled with adatabase 454 storing selected proprietary data and executable code forperforming the web application. The driver behavior monitoring computersystem 400 is operative to selectively transmit data to the respectivedatabases 450, 452, 454 through Internet 428, ISP 426, local network 422and communication interface 418, and/or to receive selected data pushedfrom the databases 450, 452, 454, or by both means in accordance withthe example embodiments. The received data is processed executed by theprocessor 404 as it is received, and/or stored in storage device 410, orother non-volatile storage for later processing or data manipulation.

Although the driver behavior monitoring computer system 400 is shown inFIG. 4 as being connectable to a set of three (3) servers, 430, 432, and434, those skilled in the art will recognize that the driver behaviormonitoring computer system 400 may establish connections to multipleadditional servers on Internet 428. Each such server in the exampleembodiments includes HTTP-based Internet applications, which may provideinformation to the driver behavior monitoring computer system 400 uponrequest in a manner consistent with the present embodiments.

Selectively locating the proprietary commercial data in database 425within the firewall 440 is advantageous for numerous reasons includingenabling rapid comprehensive local queries without substantial networkoverhead. However, it is important to maintain the accuracy of the databy performing update or refresh operations on a schedule based on thecharacteristics of the desired data or on the data requirements of aparticular query.

The driver behavior monitoring computer system 400 suitably includesseveral subsystems or modules to perform the direct and/or indirectdriver behavior monitoring as set forth herein. A primary purpose of thesubject application is to provide improved monitoring of driver behaviorwhich allows fleet managers or the like to better manage their driveroperators. In this regard, FIG. 4a is a block diagram that illustratesexecutable logic components of the driver behavior monitoring systemhaving a driver facing camera according to the example embodiment. Withreference now to that Figure, logic stored in the storage device 410(FIG. 4) is executable by the processor to perform the driver behaviormonitoring and reporting in accordance with the embodiments herein. Thelogic stored in the storage device 410 includes control logic 460control logic stored in the non-transient memory device. The controllogic is executable by the processor to process image data to determinean operational value of a parameter of a monitored condition of theassociated vehicle, perform a comparison between a recommended valuerange of the parameter of the monitored condition of the associatedvehicle and the operational value of the parameter of the monitoredcondition of the associated vehicle, and determine a state of vehicleoperation compliance in accordance with a result of the comparisonbetween the recommended value range and the operational value of theparameter of the monitored condition of the associated vehicle. Theprocessor of the system may selectively generate result data inaccordance with the result.

The logic stored in the storage device 410 further includes facialdetection logic 462 stored in the non-transient memory device. Thefacial detection logic is executable by the processor to process imagedata to locate one or more face candidate areas of an image captured bythe imaging device 345 likely above a predetermined threshold stored inthe non-transient memory device of the system to be representative of acorresponding one or more human faces in the associated vehicle, andgenerate a set of face descriptors for each of the one or more facecandidate areas. The facial detection logic is further operable toprocess the image data to determine an identify of a human personassociated with the set of face descriptors for each of the one or moreface candidate areas.

The logic stored in the storage device 410 further includes voicedetection logic 464. The voice detection logic 464 is executable by theprocessor to identify of a human person associated with a set of facedescriptors for each of one or more face candidate areas in accordancewith received voice data representative of a recorded voice of one ormore human passengers corresponding to the one or more face candidateareas.

The logic stored in the storage device 410 further includes mouthmovement logic 466. The mouth movement logic 466 is executable by theprocessor to identify of a human person associated with a set of facedescriptors for each of one or more face candidate areas in accordancewith voice data in combination with received mouth movement datarepresentative of recorded mouth movement images of one or more humanpassengers corresponding to the one or more face candidate areas.

The logic stored in the storage device 410 further includes driver headdetection logic 468. The driver head detection logic 468 is executableby the processor to process image data to locate/determine a headcandidate area of an image captured by the imaging device 345 likelyabove a predetermined threshold stored in the non-transient memorydevice to be representative of a head of an associated driver disposedin the associated vehicle, and tag a portion of the image datacorresponding to the head candidate area located/determined by thedriver head detection logic as driver head image data.

The logic stored in the storage device 410 further includes driver headdirection logic 470. The driver head direction logic is executable bythe processor to process driver head image data to determine a facingdirection of a head of an associated driver, and generate driver headfacing direction data, the driver head facing direction data beingrepresentative of the determined facing direction of the head of theassociated driver.

The logic stored in the storage device 410 further includes driver headlocation logic 472. The driver head location logic is executable by theprocessor to process driver head image data together with vehiclegeometry data and imaging device position data to determine a locationof a driver's head relative to one or more controls structures of anassociated vehicle, and generate driver's head location data, thedriver's head location data being representative of the determinedlocation of the head of the associated driver relative to the one ormore controls structures of the associated vehicle.

The logic stored in the storage device 410 further includes driver facedetection logic 474. The driver face detection logic is executable bythe processor to process image data together with vehicle geometry dataand imaging device position data to determine one or more foregroundobjects in the image data and one or more background objects in theimage data. The determined one or more foreground objects in the imagedata are disposed in the associated vehicle between the imaging deviceand the one or more background objects in the image data. The driverface detection logic is executable by the processor to process a portionof the image data corresponding to the determined one or more foregroundobjects in the image data to selectively determine, from the image data,a face of the driver of the associated vehicle, and generate a one of:driver's facial characteristic data representative of the selectivelydetermined face of the associated driver, or impeded image datarepresentative of an inability of the driver face detection locationlogic to selectively determine the face of the driver of the associatedvehicle from the image data. The driver face detection logic is furtherexecutable by the processor to process the driver's head location dataand a facial normal vector to selectively determine, from the imagedata, a face of the driver of the associated vehicle, and generate a oneof: driver's facial characteristic data representative of theselectively determined face of the associated driver, or impeded imagedata representative of an inability of the driver face detectionlocation logic to selectively determine the face of the driver of theassociated vehicle from the image data.

The driver facing camera 345 of the example embodiment is, preferably, adriver facing video camera 510 disposed as shown in FIG. 5a at the uppertop of the windshield 512 of the associated vehicle. In that position,the driver facing video camera (DFC) 510 is best able to image the head520 of the driver, and the area 530 surrounding the driver while alsosimultaneously giving an advantageous view of the road ahead for theforward facing camera. An alternative embodiment with separatedriver-facing 345 and forward-facing 346 cameras is possible, in whichcase the forward facing camera (FFC) 346 is best placed high on thewindshield as shown, and the driver facing camera 345 may be disposed ina separate housing and placed ahead on the dashboard or to the side ofthe driver, either low on the dashboard or high on the windshield asshown. Applicable unimpeded view requirements for vehicles are typicallyfulfilled by these locations. A central point of view is best forobtaining a full cabin image. In accordance with embodiments herein, oneor more still and/or video images of the driver's head are used todirectly monitor the driver behavior in ways to be described in greaterdetail below and, correspondingly, in accordance with embodimentsherein, one or more still and/or video images of the area 530surrounding the driver are used to directly monitor the driver behaviorin ways to be described in greater detail below.

FIG. 5b is a diagram showing the driver facing video camera 510 inaccordance with an example embodiment herein. As shown, the driverfacing video camera 510 includes a housing member 512 supporting a pairof first 540 and second 542 lights disposed on opposite sides of acentrally located camera device 550. The pair of first and second lights540, 542 are, preferably infrared (IR) lights, such as IR LEDs, so thatthe driver and the area in the vehicle surrounding the driver may beilluminated for purposes of recording images of the drier and the areassurrounding the driver by the camera device 550 without impeding thedriver during operation of the vehicle such as by distracting orblinding the driver, or the like. The camera 550 is preferably angledsomewhat toward the driver in order that the beneficial opticalcharacteristics, such as higher resolution, near the central axis of thelens are favored. In the embodiment, the lens horizontal field of viewis wide enough to see both the driver and passenger. The lens horizontalfield of view further is wide enough to see the driver, anypassenger(s), and the inside of the cab of the vehicle to a large extentincluding for example the vehicle side view mirrors as will be describedin detail below.

FIG. 6a is a calibration image 600 obtained from the driver facingcamera 345 showing an image of a driver 610, an image of a driver's seat620 with the driver disposed thereon, an image of a properly wornseatbelt 630, an image of a passenger side mirror 640, and an image of adriver's side view mirror 650. The calibration image 600 may be obtainedby imaging a human driver properly located in the seat, with theseatbelt being properly worn and with the driver's head being disposedin a direction to look directly at the road ahead. In the embodimentsherein, one or more portions of the calibration image 600 may be usedfor monitoring the driver's behavior directly using the driver facingcamera 345 in accordance with a detected head position of the driverwithin the vehicle being operated by the vehicle, and for monitoring thedriver's behavior indirectly using the driver facing camera 345 inaccordance with detected aspects of components of the interior of thevehicle being operated by the vehicle such as, for example, detectedaspects of the driver's seat 620, the seatbelt 630, the left and rightside view mirrors 640, 650 and other things including the absence of anypassengers in the calibration image 600. In accordance with theembodiments, the calibration image 600 may be obtained by imaging ahuman driver properly located in the seat while the vehicle is moving athigher speeds such as, for example, over 40 mph, during which driverhead pose data may be collected, thereby determining the driver's head“straight ahead” disposition. It may be assumed in the embodiment thatthe average or most common (mode) of driver's head angles correspond tothe ‘looking straight ahead, at the road’ values for this driver. It isto be noted that a yaw angle of zero may be taken as either lookingdirectly at the camera, so a frontal view of the driver, or may be takenas when looking straight ahead, that is, (typically) in line with thelongitudinal axis of the driver's seat, so facing forward, and the road.

FIG. 6b is a calibration image 602 obtained from the driver facingcamera 345 showing an image of the driver 610, an image of a driver'sseat 620 with the driver disposed thereon, an image of an improperlyworn seatbelt 630′, an image of a passenger side mirror 640, and animage of a driver's side view mirror 650. The calibration image 602 maybe obtained by positioning the human driver in the seat, with theseatbelt improperly (not) worn and with the driver's head being disposedin a direction to look directly at the road ahead. In the embodimentsherein, one or more portions of the calibration image 602 may be usedfor monitoring the driver's behavior directly using the driver facingcamera 345 in accordance with a detected head position of the driverwithin the vehicle being operated by the vehicle, and for monitoring thedriver's behavior indirectly using the driver facing camera 345 inaccordance with detected aspects of components of the interior of thevehicle being operated by the vehicle such as, for example, detectedaspects of the driver's seat 620, the improperly worn seatbelt, theseatbelt buckle 632, the left and right side view mirrors 640, 650 andother things including the absence of any passengers in the calibrationimage 602.

FIG. 7 is an example of an image 700 obtained from the driver facingcamera 345 during operation of the vehicle such as, for example, whilethe vehicle is being driven, showing an image of the driver 710, animage of the driver's seat 720 with the driver disposed thereon, animage of the seatbelt 730, an image of the passenger side mirror 740,and an image of the driver's side view mirror 750. The image 700 is inaccordance with an embodiment herein, obtained continuously as a videowhile the associated vehicle is being driven by the driver and storedinto the memory device as video data. The image 700 may also be obtainedcontinuously as a sequence of photo images taken over time and atpredetermined intervals selected for example based on the speed or otheroperational characteristics of the vehicle while it is being driven bythe driver, and stored into the memory device 340 as sequenced photoimage data. In the embodiments herein, one or more portions of the image700 may be used for monitoring the driver's behavior directly using thedriver facing camera 345 in accordance with a detected head position ofthe driver within the vehicle being operated by the vehicle, and formonitoring the driver's behavior indirectly using the driver facingcamera 345 in accordance with detected aspects of components of theinterior of the vehicle being operated by the vehicle such as, forexample, detected aspects of the driver's seat 720, the improperly wornseatbelt, the seatbelt buckle 732, the left and right side view mirrors740, 750 and other things including the presence of any passengers 760,762, and 764 in the image 700.

As noted above, in embodiments herein, systems and methods are providedusing the driver facing camera 345 for monitoring driver behaviordirectly in accordance with a detected head position of the driverwithin the vehicle being operated by the driver. The driver behaviorbeing monitored includes, in the various embodiments, one or more of:

-   -   1) a verification of a proper usage by the driver of the        driver's side view mirror 750 and/or of the passenger's side        view mirror 740;    -   2) a verification of proper attention being paid by the driver        to the road ahead;    -   3) a verification of the driver not excessively reaching for        items beyond his considered to be safe grasp space, preferably        an extent of a reach maneuver capable of being performed by the        driver without excessive body movement; and    -   4) a verification of a driver's head pose distribution metric.

The verification of the proper usage by the driver of the driver's sideview mirror 750 and/or of the passenger's side view mirror 740, of theproper attention being paid by the driver to the road ahead, of thedriver not excessively reaching for items beyond his considered to besafe wingspan, and of the driver's head pose distribution metric may besingularly and/or collectively reported to an associated fleetmanagement network, stored locally, or any combination of remotesingular/collective reporting and/or local storing. The verification ofthe proper attention being paid by the driver to the road ahead is usedin an embodiment to adapt a Lane Departure Warning (LDW) system to adetermined driver road attention value.

In further embodiments herein and as noted above, systems and methodsare provided using the driver facing camera 345 for monitoring driverbehavior indirectly in accordance with detected aspects of components ofthe interior of the vehicle being operated by the driver. The driverbehavior being monitored includes in the various embodiments one or moreof:

-   -   1) a verification of a proper usage by the driver of a seatbelt;    -   2) a verification of the driver having proper hand placement on        the steering wheel; and    -   3) a verification that the driver has either no passengers, a        proper limit of passengers, and/or a verification that the        detected passengers are authorized passengers.

Using Forward Facing Camera (DFC) to Monitor and Report Driver Behavior

As noted above, the example embodiments herein are provided formonitoring and reporting driver behavior directly using a driver-facingcamera in accordance with a detected head position of the driver withinthe vehicle being operated by the driver, and for monitoring andreporting driver behavior indirectly using a driver-facing camera inaccordance with detected aspects of components of the interior of thevehicle being operated by the driver. In the direct driver behaviormonitoring, the driver and/or the driver's head is located in the imageobtained of the vehicle interior, and parameters of various driverbehavior metrics are determined in accordance with the located driverhead in the image. In the indirect driver behavior monitoring, one ormore components of the vehicle such as for example a seat belt or asteering wheel are located in the image obtained of the vehicleinterior, and parameters of various driver behavior metrics aredetermined by inference in accordance with the located one or morecomponents of the vehicle in the image.

FIG. 8 is a flow diagram showing a method 800 of implementing a driverbehavior monitoring and reporting strategy in accordance with an exampleembodiment including a first set of steps 820 for monitoring driverbehavior indirectly using a driver-facing camera in accordance withdetected aspects of components of the interior of the vehicle beingoperated by the vehicle, and further including a second set of steps 830for monitoring driver behavior directly using the driver-facing camerain accordance with a detected head position of the driver within thevehicle being operated by the vehicle. In the first set of steps 820indirectly monitoring the driver behavior, vehicle cabin image data iscollected and then analyzed at step 822. In the embodiment, the vehiclecabin image data is representative of the image 700 (FIG. 7) obtainedfrom the driver facing camera 345 during operation of the vehicle.Thereafter, one or more action(s) are taken in step 824 based on thecollected and analyzed cabin image data. In the embodiments described,the indirect driver behavior monitoring does not rely on finding thelocation, position or pose of the driver's head in the image, but ratherinfers the driver's behavior from portions the image relating tocomponents of the vehicle being used by the driver, preferably beingused in accordance with a good driver behavior such as for example aproper wearing of seatbelts.

Somewhat similarly in the second set of steps 830 directly monitoringthe driver behavior, a portion of the vehicle cabin image data relatingto the vehicle driver image is segregated at step 832 from the vehiclecabin image data collected at step 822. The segregated portion may berelated to the driver's head, the driver's seat, the seatbelt, theseatbelt buckle, the one or more passengers, or any other items selectedfor monitoring as may be necessary and/or desired. Thereafter, one ormore action(s) are taken in step 834 based on the vehicle driver imageportion of the cabin image data.

I. Using DFC to Indirectly Monitor and Report Driver Behavior

Driver behavior may be monitored, in accordance with embodimentsdescribed herein by using a driver-facing camera to detect and monitoraspects of components of the interior of the vehicle being operated bythe vehicle, then inferring driver behavior in accordance with themonitored aspects of components of the interior of the vehicle. Theindirectly monitored driver behavior is collected and stored locally inthe vehicle and, in embodiments, may be reported to a central fleetmanagement system.

Passenger Detection And Counting

Commercial vehicle drivers may have one or more unauthorized passengersaccompanying the driver in the vehicle. Commercial vehicle fleet policyoften forbids or limits the passengers allowed to be present in theirvehicles. It would therefore be desirable to detect if any unauthorizedpassengers are present in the vehicle. It would also be desirable todetect how many passengers are present in the vehicle. It would furtherbe desirable to identify the detected passengers present in the vehicle.

The example embodiment as shown for example in FIG. 9 provides a systemand method for detecting, counting, and identifying such passengers. Anadvantage of the example embodiment is an ability to enforce fleetpolicy by ensuring that the driver adheres to fleet policy, and that anyfleet policy violations are appropriately logged and reported.

In the embodiment of the method 900 shown in FIG. 9, the cabin imagedata collection portion 822′ includes a step 902 determining a time ofthe image of the cabin, and a step 904 collecting vehicle operationaldata such as, for example, vehicle speed data or the like. In step 906the logic of the system finds one or more faces in the cabin image data,and further, counts the number of faces found. In step 908 of the cabinimage data collection portion 822′, the logic of the system is executedto attempt to identify the one or more faces found in the cabin imagedata.

Next in the method 900 shown in FIG. 9, the action taking portion 824′includes a step 910 of determining whether any of the faces located inthe cabin image data can be or have been identified. If one or more ofthe faces are identified, the method 900 in step 920 stores anidentification of the faces together with the vehicle status datacollected in step 904. On the other hand, If any of the faces are notidentified, the method 900 in step 930 stores the determined face counttogether with the vehicle status data collected in step 904.

Further in the method 900 of the embodiment, one or more of theidentification of the faces, the determined face count, and/or thevehicle status data is stored locally in the memory of the system at thevehicle or is transmitted in step 940 to a central fleet managementsystem.

In accordance with the example embodiment, the driver facing camera 345uses wide angle camera views to obtain an image 700 of the cabin of thecommercial vehicle. This wide angle image is preferably then undistortedto remove wide angle lens effects. The undistorted cabin image data isinspected by the logic 330 to first locate faces in the image, and thento count the located faces. Face detection algorithms, such as that ofViola-Jones, may be used to locate candidate camera image areas that maybe faces. Face descriptors are generated for these located facecandidate camera image areas. The number of detected face areas,overlapping and not, and the corresponding face descriptors aregenerated. A threshold for facial similarity is set, below which thefaces are declared to be the same (via similar face descriptor vectors).Similarly, the detected faces may be compared with previously storedface descriptor vector data for drivers and passengers allowed to be inthe vehicle. The face descriptor vector data of authorized drivers andpermitted passengers may be stored locally in the driver behaviormonitoring system or remotely such in one or more databases associatedwith the servers 142 (FIG. 1), of the central fleet management system.

Tracking logic executed by the processor may be used to associate facialmeasurements with previous locations, thereby allowing personidentification logic executed by the processor with a focus on multipleareas. The identified (or not) persons are transmitted to the one ormore Fleet Management servers 142 (FIG. 1), together with vehicle statedata preferably sampled coincidently with person(s) identification. Thismay occur either while the vehicle is moving or while it is stationaryor standing still.

The identified faces are compared with either an in-vehicle database, ortransmitted to a central management system 142 with a similar database150 (FIG. 1). Should a person not registered as allowed in the vehiclebe identified, a first pass is made at identifying said person(s). Ifthe identified one or more person(s) is/are known to the database, afirst type of event processing is performed by the driver behaviormonitoring computer system. However, if the identified one or moreperson(s) is/are unknown to the database, a second type of eventprocessing is performed by the driver behavior monitoring computersystem.

The information is selectively transmitted to the fleet managementsystem for analysis by a fleet manager 160 (FIG. 1) or the like. Theinformation collected, analyzed, and transmitted may include any one ormore or others of: how many passenger(s) (i.e. not driving) are presentin a vehicle, whether these passengers are known or not, facialdescriptors may be sent to an associated fleet management system if theidentified passengers are not known, the gender of the passengers, atime of day of image collection, a location of the vehicle at the timethe cabin image was collected, passenger snapshot(s), and vehicleinterior/cabin snapshot(s) as may be deemed necessary and/or desired.Unknown passengers may also be recorded by a microphone of the inputdevice 414 (FIG. 4) which may be present in the system when it isdetermined that the passengers are speaking.

FIG. 9a shows a further method 950 for detecting if any unauthorizedpassengers are present in the vehicle, how many passengers are presentin the vehicle, and the identities of any detected passengers present inthe vehicle. In the embodiment, the method 950 includes a series ofsteps that determine when passenger detection is performed, and what isdetected and sent. Passenger visibility may be typically associated withthe use of the passenger door opening and closing. In the example,passenger detection is performed only in response to selectabletriggering events and is otherwise not performed. In the embodiment, atemplate image of the passenger door in opened, closed, opening,closing, and ajar conditions is used to detect the status of thepassenger door as being opened, closed, uncertain, or the like. FIGS.6a, 6b , and 7 for example show the driver door (similar appearance tothe passenger door) beyond the driver, and it is the edges of this, at afixed location, which is used by the system in accordance with themethod 950 to determine whether it is open or closed.

The method 950 is initiated at step 952 by the system of an example theembodiment wherein a number of circumstances or trigger events aredetermined at step 954 for moving forward with the method 950 fordetermining whether any passengers are in the vehicle. If none of thetriggering events are detected at step 954, the passenger detectionmodule is not executed. However, the occurrence of any one or more ofthe trigger events being detected at step 754 will lead to the passengerdetection module execution. In the example embodiment, the triggerevents may include any one or more of the door being just (recently)opened, and the vehicle has recently stopped; the door being just(recently) closed (at which point an image is stored) and the vehiclebegins moving thereafter; when the vehicle has just started movingforward; when a predetermined time for execution arrives, such as amonitoring interval; when a stop has occurred at an unusual location,such as on a highway, and the passenger door is open. Other triggerevents may be used and are contemplated in the embodiment. A black boxtype data storage scheme may be used to retrieve suitable passengerimages prior to the door being opened or just after it is closed.Suitability may be determined by the finding of a face oriented forwardtoward the windshield in the location where a passenger would appear.

When such circumstances occur, an image of the cabin is made in step 956using the driver facing camera 345 of the embodiment described above.All faces in this image are located at step 958 by the logic of thesystem. Facial descriptors are generated in step 960 for these faces.The descriptors are compared in step 962 with one or more of anon-vehicle database 340 (FIG. 3) and/or off-vehicle database 450, 452,454 (FIG. 4), and each face is labeled in accordance with thecomparison(s) as “known” or “unknown” or, alternatively, labeled asbeing “allowed” or “not allowed.” Any other suitable labels may be usedas necessary or desired.

Vehicle status information is collected and stored at step 964 and apassenger detection status report is then in step 966 stored and/or sentto a central database. This report contains one or more of how manypeople are present in the vehicle, their identities (with the unknownJohn or Jane Doe state also possible), the cabin image, the vehiclelocation, the vehicle speed, the door status(plural possibly), theforward view, an audio recording if speech is detected from themicrophone or lip motion signals.

A system is provided for monitoring a permitted occupant condition of anassociated vehicle during operation of the associated vehicle by anassociated driver. The system includes an imaging device disposed in theassociated vehicle, a control device, facial detection logic, andcontrol logic. The imaging device captures an image of the associateddriver disposed in the associated vehicle. The imaging device alsocaptures an image of an interior of the associated vehicle, andgenerating image data representative of the captured image of theassociated driver disposed in the associated vehicle and of the interiorof the associated vehicle. The control device includes a processor, animage data input operatively coupled with the processor, and anon-transient memory device operatively coupled with the processor. Theimage data input is configured to receive the image data from theimaging device. The facial detection logic is stored in thenon-transient memory device, and is executable by the processor toprocess the image data to locate one or more face candidate areas of theimage captured by the imaging device likely above a predeterminedthreshold stored in the non-transient memory device to be representativeof a corresponding one or more human faces in the associated vehicle.The facial detection logic is further executable by the processor togenerate a set of face descriptors for each of the one or more facecandidate areas. The control logic stored is also stored in thenon-transient memory device and is executable by the processor todetermine, based on the set of face descriptors generated for each ofthe one or more face candidate areas, a vehicle occupant count as anoperational value of an occupant quantity parameter of the monitoredpermitted occupant condition of the associated vehicle. The vehicleoccupant count may be stored locally in the memory of the vehicle and/ortransmitted to the central fleet management system.

Calibrated Seat Belt Usage Detection System

Too many drivers fail to regularly wear their seat belt therebycompromising their own personal safety. For commercial vehicle drivers,however, not wearing a seat belt may also violate fleet policy.

It is therefore desirable to detect whether or not a driver is properlywearing her/his seat belt during vehicle operation. In this regard, beltusage detection systems, methods, and apparatus are provided asdescribed below.

Cameras are becoming somewhat ubiquitous in commercial vehicles forrecording in a digital “loop” a video of the roadway ahead of thevehicles as they travel. The video is useful for accident recreationpurposes and for other memorializing of the most recent activities ofthe vehicle and driver should any mechanical or other issues arise.Driver facing cameras have been used as well for imaging the driver fromtime to time as necessary such as, for example, whenever the vehicle isstarted so that the identity of the person in control of the vehicle canbe determined at a later time.

In further embodiments herein, camera-based systems, method, andapparatus are provided for detecting whether a seat belt is being worn.An example embodiment of a method for detecting whether a seat belt isbeing worn is shown in FIGS. 10, 10 a, and 10 b. Expected features of aworn seat belt are sought in an image 700 (FIG. 7) taken by the driverfacing camera 345. These features may include lines emanating from anorigin point or region within a predetermined portion of the image 700.Alternatively or in addition, these features may include lines in theimage within a range of angles. Alternatively or in addition, thesefeatures may including lines in the image with a range of colors betweenthe lines, without discontinuity or if a discontinuity is present, wherethe line ends near the discontinuity point approximately parallel to andat each other.

FIG. 10 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a seatbelt usage detection, monitoring, and reportingstrategy in accordance with an example embodiment. With reference now tothat Figure, in the method 1000 of the embodiment, the cabin image datacollection portion 822′ includes a step 1012 determining a time of theimage of the cabin, and a step 1014 collecting vehicle operational datasuch as, for example, vehicle speed data or the like. In step 1016 thelogic of the system finds a seatbelt origin point in the cabin imagedata, and further, determines a disposition of the seatbelt in step1018.

Next in the method 1000 shown in FIG. 10, the action taking portion 824′includes a step 1010 of determining whether the driver's seatbelt isproperly worn. If the driver's seatbelt is properly worn, the method1000 in step 1020 stores an “ON” identification or “ON” seatbelt statusdata. The “ON” identification or “ON” seatbelt status data may be storedtogether with the image of the cabin collected at step 1014 as may benecessary and/or desired. On the other hand, if the driver's seatbelt isnot properly worn, the method 1000 in step 1030 stores the an “OFF”identification or “OFF” seatbelt status data. Similar to the “ON”identification above, the “OFF” identification or “OFF” seatbelt statusdata may be stored together with the image of the cabin collected atstep 1014 as may be necessary and/or desired.

Further in the method 1000 of the embodiment, one or more of the “ON”identification or “ON” seatbelt status data, the “OFF” identification or“OFF” seatbelt status data, and/or the image of the cabin collected atstep 1014 is stored locally in the memory of the system at the vehicleor is transmitted in step 1040 to a central fleet management system.

FIG. 10a is a flow diagram showing details of a portion of the method ofoperating a driver behavior monitoring system having a driver facingcamera for implementing the seatbelt usage detection, monitoring, andreporting strategy of FIG. 10, in accordance with an example embodiment.With reference now to that Figure, a calibration image 600 (FIG. 6a ) ofthe driver 610 properly wearing his seatbelt is retrieved at step 1050from a local memory of the system. The calibration image may be obtainedin an initial step where the driver is requested to first not wear hisseatbelt, and then in a second step, to wear his seatbelt.Alternatively, a generic model of a properly worn seatbelt 630 isretrieved at step 1050 from the local memory. The image of the vehiclecabin obtained at step 1014 (FIG. 10) is compared in step 1052 againstthe calibration image 600 and/or against the generic model of a properlyworn seatbelt 630.

In step 1054 the system determines whether a seatbelt is “seen” orotherwise detected in the image of the vehicle cabin obtained at step1014. If a seatbelt is seen at step 1054, the system concludes at step1056 that the driver is indeed properly wearing his seatbelt. The methodflow returns thereafter to the action taking portion 824′ (FIG. 10) ofthe method of operating a driver behavior monitoring system inaccordance with the embodiment. However, if a seatbelt is not seen atstep 1054, a second examination at step 1058 is performed for thelightness or darkness of the driver's body covering (below the head). Ifthis area is dark, it is possible that the driver is wearing darkclothing against which the seat belt may not be seen. If the driver iswearing dark clothing, then no judgement may be made regarding whetherhe is properly using the seat belt and the system concludes at step 1060that the driver is properly wearing his seatbelt. If light clothing isdetected, and no seat belt is seen the system concludes at step 1060that the driver is not properly wearing his seatbelt. The method flowreturns thereafter to the action taking portion 824′ (FIG. 10) of themethod of operating a driver behavior monitoring system in accordancewith the embodiment.

FIG. 10b is a flow diagram showing further details of a portion of themethod of operating a driver behavior monitoring system having a driverfacing camera for implementing the seatbelt usage detection, monitoring,and reporting strategy of FIG. 10, in accordance with an exampleembodiment. With reference now to that Figure, a calibration image 602(FIG. 6b ) of the driver 610 improperly wearing his seatbelt isretrieved at step 1070 from a local memory of the system. Alternatively,a generic model of an improperly worn seatbelt 630′ is retrieved at step1070 from the local memory. The image of the vehicle cabin obtained atstep 1014 (FIG. 10) is compared in step 1072 against the calibrationimage 602 and/or against the generic model of a properly worn seatbelt630′.

In step 1074 the system determines whether a buckle 631′ of an unbuckledseatbelt is “seen” or otherwise detected in the image of the vehiclecabin obtained at step 1014. If a buckle 631′ of an unbuckled seatbeltis not seen at step 1074, the system concludes at step 1076 that thedriver is wearing a jacket or the like. The method flow returnsthereafter to the action taking portion 824′ (FIG. 10) of the method ofoperating a driver behavior monitoring system in accordance with theembodiment. However, if the buckle 631′ of an unbuckled seatbelt is“seen” or otherwise detected at step 1074 in the image of the vehiclecabin obtained at step 1014, the system concludes at step 1078 that thedriver is not wearing a jacket. The method flow returns thereafter tothe action taking portion 824′ (FIG. 10) of the method of operating adriver behavior monitoring system in accordance with the embodiment.

In an embodiment, a calibration image or model of seat belt appearanceis taken or established. A matched model of the seat belt buckle may beapplied to where the seat belt buckle may be visible. That is, a buckle632 (FIG. 6b ) should not be visible near the origin of the seat beltover the driver's shoulder. A warning or other action or function may beissued or otherwise started upon detection of the seat belt not beingworn by the driver.

The driver facing camera 345 obtains an image 700 (FIG. 7) duringoperation of the vehicle and, in this way, the camera may see or knowthe origin point/region for the seat belt, which may be used to detectwhether the seat belt is worn. FIG. 7 shows a user wearing her seatbelt. These cameras see the origin point for the seat belt, and whetherthe seat belt is worn. The example embodiment advantageously usesknowledge of the origin point of the seatbelt, together with acalibration image 600 (FIG. 6a ) of a driver 610 wearing the belt 630,or a generic model of seat belt appearance (angle, width, originlocation, end location) in the image, to detect parallel lines withinthe appropriate width range, and originating and ending where expected.If no belt is seen, the method of the example embodiment is configureddetermine whether the driver's jacket is dark, thus rendering a darkbelt invisible, for example. In that case, the method first triesincreasing line detection sensitivity, failing which the methoddeclares, by a benefit of the doubt analysis, that the driver is wearinga belt. If a lighter upper garment is worn and no dark (dark relative tothe light upper garment) belt is detected, the method of the embodimentgenerates a signal that a lighter upper garment is worn and no belt isdetected for storage in the local memory and/or for transmission to thecentral fleet management system.

It is to be understood that the seat belt is visible as a differentlycolored (contrasting) band-shaped area, with contrast to the objectsnext to or behind it. Where the seat belt is obscured by the person'sscarf or face, a front edge line may still be visible and continuesupward to reconnect to the ‘two-sided’ segment. Even when the seat beltis obscured by the full extent of the driver's clothing or the like, theends would still be visible and would continue and ‘point at’ each otherapproximately. It is to further be noted that the seat belt is to theleft of (in the image)/in front of the no person present location, wereit buckled. The system thus has an expectation of what the image of thebelt, properly worn, should looks like: (parallel/single/perhapspartially or fully obscured) lines, running in an approximate direction,between two known points (regions), and within a certain portion of theimage. The system has the further expectation of what the visibleportion of the belt looks like when worn behind the user. In thisregard, the diagonal edges of the seat belt may advantageously bedetected in accordance with the embodiments herein using, for example,Kirsch or other equivalent edge filters or the like.

In accordance with the embodiments herein, the system is not fooled orotherwise tricked into a determination of good seatbelt usage behaviorby a driver wearing a ‘seat belt t-shirt’ (a shirt having a diagonaldark stripe graphic that appears to be a seat belt being worn). In thisembodiment, the system inspects the cabin image for a set of nearlyparallel edges emanating from the seat belt upper anchor point. Inanother embodiment, the system inspects the cabin image for linescontinuing beyond the ‘seat belt’ (the false printed seat belt) that thedriver appears to be wearing. Even if the user buckles the belt behindherself, a discontinuity is observed or otherwise detected by the systembetween the actual physical belt and the false belt pattern printed inthe t-shirt. The system, by looking for this break, is able to detectthat the user driver is not properly using the seatbelt.

Using knowledge of the origin point (or range), together with thecalibration image 600 (FIG. 6a ) of a driver 610 wearing the belt 630,or a generic model of seat belt appearance (angle, width, origin, end)in the image, also without a user present, the system is configured todetect parallel lines within the appropriate width range, andoriginating and ending where expected. If no belt is seen, the systemchecks if the driver's jacket is dark (thus rendering a dark beltinvisible, for example), in which case it first tries increasing linedetection sensitivity, failing which the system declares, by a benefitof the doubt analysis, that the driver is wearing a belt. If a lighterupper garment is worn and no belt is detected, the system signals this.

Alternatively, the system may detect the (generally shiny, and thereforeprobably light and contrasting) possibly visible buckle of the seat beltif the belt is not worn and not buckled. The camera is, either fromknown geometric installation values, or in a calibration step (simplysignal where the belt origin point is), knows/is taught where the bucklewould be visible. If the seat belt is perhaps not being worn, the systemmay switch to this second mode and detect the presence of the(unlatched) buckle 632 at the origin point such as shown, for example,in FIG. 6b . The origin point is furthermore typically fixed or conformsto a linear set of locations in the image. In an embodiment, a fixedpatch is defined in the calibration image 602 (FIG. 6b ) of the imagewhere an unworn seat belt buckle 631′ must appear, and it is there thatthe system may search for the buckle. If the seatbelt buckle is found inthis fixed patch area, the system concludes that the driver is notwearing the seatbelt. Equivalently, for each driver, there is a fixedpatch of the image corresponding to where a properly worn buckle 631(FIG. 6a ) appears. A matched template set of such properly buckled andunbuckled images appears may be stored and compared by the system withthe actual image. Sufficient correspondence between an image in thestored sets and the DFC image patch corresponding to where the bucklemay be leads the system of the embodiment to conclude that the driver iswearing, or not, her seatbelt.

A system is provided for monitoring seatbelt usage by a driver of avehicle during operation of associated vehicle by the driver. The systemincludes an imaging device, a non-transient memory device storing safemodel data comprising a recommended value range of a seatbelt useparameter of a monitored seatbelt worn by the associated drivercondition of the associated vehicle, control logic stored in thenon-transient memory device, and an output. The imaging device capturesan image of an interior of the associated vehicle together with an imageof the associated driver disposed in the associated vehicle, andgenerates image data representative of the captured images of theassociated driver and the interior of the associated vehicle. Thecontrol logic is executable by the processor to process the image datato determine an operational value of the seatbelt use parameter of themonitored seatbelt worn condition of the associated vehicle, perform acomparison between the recommended value range of the seatbelt useparameter of the monitored seatbelt worn condition of the associatedvehicle and the operational value of the seatbelt use parameter of themonitored seatbelt worn condition of the associated vehicle, anddetermine a state of vehicle operation compliance as a one of either aseatbelt non-compliance state or a seatbelt compliance state inaccordance with the result of the comparison.

In an embodiment, the non-transient memory device stores a calibrationimage of a driver wearing a seatbelt having an origin point relative tothe image of the interior of the associated vehicle as the safe modeldata comprising the recommended value range of the seatbelt useparameter of the monitored seatbelt worn condition of the associatedvehicle. Also in the embodiment, the control logic stored in thenon-transient memory device is executable by the processor to processthe image data to determine, based on the calibration image having theorigin point, a disposition of a seatbelt in the image data as theoperational value of the seatbelt use parameter of the monitoredseatbelt worn condition of the associated vehicle.

In a further embodiment, the non-transient memory device stores ageneric model of a physical appearance of a buckled seatbelt as the safemodel data comprising the recommended value range of the seatbelt useparameter of the monitored seatbelt worn condition of the associatedvehicle. Also in the embodiment, the control logic stored in thenon-transient memory device is executable by the processor to processthe image data to determine, based on the generic model of the physicalappearance of a buckled seatbelt, a disposition of a seatbelt in theimage data as the operational value of the seatbelt use parameter of themonitored seatbelt worn condition of the associated vehicle.

The system of the example embodiment distinguishes between the type ofnon-usage of the seat belt. These types may include, for example,buckled behind the driver (or passenger), wearing an ‘I am wearing aseat belt’ upper outer garment, or simply not wearing the belt at all.Data relating to the type of non-wearing is stored locally and/ortransmitted to the central fleet management server, along with aphotograph of the non-wearing person or persons in the vehicle.

Driver's Hands On The Steering Wheel Detection

Too many vehicle operators fail to regularly properly place their handson the steering wheel while driving, thereby compromising their ownpersonal safety and risking damage to the vehicle. For commercialvehicle drivers, however, improper, inconsistent, or lax steering wheelhand placement may also violate fleet policy.

It is therefore desirable to detect whether or not a driver has properlyplaced hands on the steering wheel during vehicle operation. In thisregard, driver's hands on the steering wheel detection systems, methods,and apparatus are provided as described below.

FIG. 11 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera forimplementing a hands on the steering wheel detection, monitoring, andreporting strategy in accordance with an example embodiment.

With reference now to that Figure, in the method 1100 of the embodiment,the cabin image data collection portion 822′ includes a step 1102determining a time of the image of the cabin, and a step 1104 collectingvehicle operational data such as, for example, vehicle speed data or thelike. In step 1106 the logic of the system finds a steering wheel shapein the cabin image data, and further, searches the cabin image data forshort (hand width dimension approximately) portions of the steeringwheel that are not visible in step 1108.

Next in the method 1100 shown in FIG. 11, the action taking portion 824′includes a step 1110 of determining whether the driver's hands areproperly on the wheel at the designated correct positions. If thedriver's hands are properly on the wheel at the designated correctpositions, the method 1100 in step 1120 stores a “Hands ON”identification or “Hands ON” steering wheel status data. The “ON”identification or “ON” steering wheel status data may be stored togetherwith the image of the cabin collected at step 1104 as may be necessaryand/or desired. On the other hand, if the driver's hands are notproperly on the wheel or are on the steering wheel but not on the wheelat the designated correct positions, the method 1100 in step 1130 storesthe a “Hands OFF” identification or “OFF” steering wheel status data.Similar to the “Hands ON” identification above, the “Hands OFF”identification or the “Hands OFF” steering wheel status data may bestored together with the image of the cabin collected at step 1104 asmay be necessary and/or desired.

Further in the method 1100 of the embodiment, one or more of the “HandsON” identification or the “Hands ON” steering wheel status data, the“Hands OFF” identification or the “Hands OFF” steering wheel statusdata, and/or the image of the cabin collected at step 1104 is storedlocally in the memory of the system at the vehicle or is transmitted instep 1140 to a central fleet management system.

FIG. 12 is an example of an image generated by the driver facing cameraof FIG. 5b and obtained by the driver behavior monitoring system duringoperation of the associated vehicle and showing a typical driver havinghis hands on the steering wheel. Official recommendations are to havethe driver's 1210 left hand 1222 at between the 9 and 10 o'clockpositions on the steering wheel, and the right hand 1220 at between the2 and 3 o'clock positions on the steering wheel, which is visible in theimage 1202 of FIG. 12. The wide spacing is recommended because of theshape of the expanding airbag, should a collision occur. In theembodiment, the system seeks to find the driver's steering wheel handpositions in the image obtained by the driver facing camera 345. Nothaving one's hands at the recommended positions, or not having bothone's hands on the wheel at all or not as frequently as may be required,is flagged by the system as a fleet policy infraction which is stored inthe local memory and/or transmitted to the central fleet managementsystem.

The embodiment takes advantage of the physical nature of steering wheelsin commercial vehicles, which are nearly always circular. Circularshapes are easily detected in images, even when seen in a skewed view.The driver facing camera 345, typically sees the both the driver andsteering wheel (if not the whole wheel, then a significant fractionthereof) which appears as an ellipse. A Hough Transform is used forellipse detection (after lens distortion is accounted for) on an edgeimage from the driver facing camera 345. Only the edge points in theoriginal image need to be undistorted, thus saving computation time. TheHough Transform returns where in the image the (elliptically shaped inthe undistorted image) steering wheel is located. Those edge pixels aremarked in the image that correspond to the wheel. Pixels in the imagerelating to unseen portions of the wheel may also be marked with indiciarepresentative of information relating to where the wheel would be inthe image, were the view of it not blocked. A model for the appearanceof the whole steering wheel is thereby provided in the image, eventhough only a segment of the steering wheel is visible in the image. Thedriver's hands and arms can obscure portions of this image as may beseen in FIG. 12.

In an embodiment, the area of the image obtained by the driver facingcamera 345 that is searched for these edge points may be limited orotherwise reduced, thereby saving processing time and improvingaccuracy. This image search area reduction may be performed based insubstantial part on a knowledge of the optical and mechanical aspects ofthe camera and of its physical installation geometry, or in an initialcalibration step, when the truck cabin's important features are located.For purposes of helping to expedite the search for items in the image,the driver facing camera 345 image edge search is constrained in theembodiment in terms of both the portion of the image to examined andwhat or which edge directions must be present or are otherwise expectedto be there in the reduced portion of the image to be examined (e.g. thesystem does not expect a vertical edge at the top of the steering wheel1230, this taken as seen from the driver's viewpoint; in the image thesteering wheel edge is in fact approximately vertical).

The Hough Transform is preferably run on the undistorted edge imageobtained from the driver facing camera 345 to detect ellipses. A highedge sensitivity may be used as necessary or desired, as the approximatelocation/appearance of the steering wheel are known, since there is onlyone ellipse, and it is within a limited size range. An alternative tothe Hough Transform is to store template images of the steering wheeland compare these with what is seen by the DFC. The steering wheelportion of these images may be identified by the Hough Transform in aninitial calibration step, and then stored, after which template matchingis performed to locate the steering wheel in an image, without needingto perform the Hough Transform again

The embodiment therefore uses knowledge of the possible steering wheellocation(-s, if adjustable), along with Hough Transform ellipsedetection, to localize the steering wheel 1230 in a driver facing cameraimage 1202 of the cabin of the vehicle. The contours of this detectedellipse 1232 are examined for missing sections 1240, 1242, indicatingthe locations where the driver's hands 1220, 1222 are, respectively, onthe wheel. That is, the hands are not directly detected; rather, theunseen portions of the steering wheel are taken as the hand location(s).

It may be seen, for instance, in FIG. 12, that the right driver's hand1220 interrupts the view of the steering wheel at 1240, but that on bothsides thereof the steering wheel 1230 may be seen. The unseen edgepixels of the steering wheel 1230 are tagged, particularly on the rightand top sides, and thus the system determines where the driver's righthand 1220 is. The left hand 1222 is where the top steering wheel sectionis no longer visible at 1242 looking to the left in the view shown. Aknowledge of the steering wheel's color can also be used in accordancewith the embodiment to help locate the steering wheel 1230 in the image1202.

In addition, the system of the embodiment may execute logic to track thedriver's hand movements relative to the steering wheel. For example, thesystem may look for active hand movement relative to the wheel (i.e. achanging hand position on the wheel), which may be used as a proxy foran attentive driver and recorded by the system as positive safetyrelated events. Episodes of a non-changing hand position on the wheelmay be used to alert the driver or may be recorded by the system asnegative safety relevant events.

In an embodiment, one or more stored template images are used fordetermining where the steering wheel may be located, should it beadjustable, in the image. The one or more stored template images arecompared with the image of the steering 1230 when obtained by thedriver-facing camera. The best matching template image effectivelylocates the wheel in the image. Following this, the ‘gap in the seen’steering wheel determination as described above is performed forlocating the positions of the driver's hands 1220, 1222 on the wheel1230 at the locations 1240, 1242 of the determined gaps in the steeringwheel image 1232.

In addition to the above, the system of the embodiment may selectivelyperform a remapping of the elliptically appearing steering wheel in theimage to a circle. This remapping corresponds to a re-projection of thewheel to a fully circular appearance. The sections of the steering wheelobscured 1240, 1242 by the driver's hands 1220, 1222 are alsoselectively transformed via this same re-mapping, and from theseremapped hand positions the driver's angular hand spacing may beascertained. Good driver behavior suggests an angular driver handspacing of between approximately 180 degrees to approximately 120degrees. The driver's hand position spacing on the wheel may be used toalert the driver or may be recorded by the system as positive ornegative safety relevant events.

Fleet management or other policy violations such as: number of hands onthe wheel, hand positions, percentage of time the driver holds thewheel, etc., may be detected, flagged, warned for, reminded about,and/or measured. Variation in hand position may be used as a proxy fordriver fatigue.

II. Using Driver Facing Camera to Indirectly Monitor and Report DriverBehavior

Driver behavior may be directly monitored, in accordance withembodiments described herein, by using an imaging device trained on thedriver while the vehicle is being operated. The monitored driverbehavior is collected and stored locally in the vehicle and, inembodiments, may be reported to a central fleet management system.

Driver's Road Attention Detection

Too many drivers fail to pay proper attention to the road ahead.Drivers' eyes often wander from being directed towards the road owing tovarious tasks to be performed while driving such as, for example,checking gauges on the instrument panel, checking for other trafficusing side view mirrors of the vehicle, operating radios or othergadgets on or in the vehicle cabin, and the like. This implies that thedriver's eyes and therefore her attention aren't always where theyshould be; namely on the road, which has the tendency of adverselyaffecting the safe operation of the vehicle, particularly when driverstake their eyes off the road for prolonged or extended period of time,or when attention is frequently directed away from the road over time.

It is therefore desirable to detect whether or not a driver is payingproper attention to the road ahead while operating the vehicle. In thisregard, driver road attention detection systems, methods, and apparatusare provided as described below.

In accordance with an embodiment, overall, the driver facing camera 345of the driver behavior monitoring system is used to detect the directionthe driver's head is facing, and the system relates this detecteddirection and the location of the driver-facing camera relative to thevehicle cabin structure to determine whether the driver is oriented suchthat the road can be properly seen. The relative position between thedriver-facing camera and the vehicle cabin structure may be based on oneor more calibration images as necessary and/or desired. The systems,methods, and apparatus of the embodiment are operable to transmit asignal to an associated central fleet management system when the driveris not oriented such that the road can be properly seen. Alternativelyand/or in addition, the systems, methods, and apparatus of theembodiment are operable to store data representative of driverinattention into a local memory device when the driver is not orientedsuch that the road can be properly seen. The locally stored driverinattention data may be downloaded when the vehicle is taken off-linefrom the road, when the vehicle is being serviced, when the driverrequests a download, or the like.

The systems, methods, and apparatus of the embodiments monitor thedriver's road attention in accordance with a combination of a locationof the driver's head and a facial normal vector of the driver's head.The location of the driver's head relative to vehicle cabin structureincluding for example the front windshield, and the facial normal vectorof the driver's head are determined by the systems, methods, andapparatus of the embodiments. This is beneficial, for example, whendrivers of different heights operating the same vehicle at differenttimes is considered. For example, a short driver will need to look upmore than a tall driver in order to properly see the road ahead.

In the example embodiment, a driver-facing camera 345 mounted on thewindshield of a vehicle views the driver 520 (FIG. 5a ) in the passengercabin 530. The image taken by the camera 345 is analyzed to find thedriver's head and which way she is facing which is expressed in theexample embodiment as a facial normal vector 522. Standard methods oflocating faces may be used for initial localization of the driver'shead, after which a shape regression is performed by the driver behaviormonitoring system to determine where the facial landmarks (e.g. nose,corners of the mouth, tragus points) are. Using these landmarks, ageneric head model is fitted, from which the facial normal vector 522 isderived, the details of which will be explained below.

A monocular camera cannot, however, determine how far away an object iswithout further information. This being the case, the driver behaviormonitoring system may determine the driver's head location in severalways, three of which will be described below.

In accordance with a first method, known landmarks on the driver's seatare used to measure the distance to and/or the height of the driver'sseat, and from these distance and/or height measurements an approximatedriver head location may be inferred. The known landmarks on thedriver's seat 620 (FIG. 6a ) are preferably contained in the calibrationimage 600 (FIG. 6a ).

In accordance with a second method, one or more calibration photos areused to determine the driver's head location. For example, the drivermay be asked to lean directly back against the fully backed seat, soproducing a known position, in the reference snapshot image 600 (FIG. 6a).

In accordance with a third method, assuming the driver 610 is sittingcentered on the seat 620 in the reference snapshot image 600 (FIG. 6a ),his nose 611 will be in the vertical half-seat plane 621, making thedriver's head 520 easy to locate in the image. Typical truck seats moveup and down, front and back, and their backrest is tiltable. The sidesof the seat therefore move within a fixed plane to some approximation. Atypical truck calibration image 600 is shown in FIG. 6a , and a typicaltruck operational image 700 is shown in FIG. 7.

The driver-facing camera 345 may locate a point 622 of the (typicallyvisible) seat side in the image such as for example at the upper leftcorner of the seatback over the driver's right shoulder or elsewheresuch as the back of the lower seat cushion just under a likely positionof a driver's ID badge on his right hip (not shown), and thereby thedriver behavior monitoring system of the embodiment establishes a ray in3-D space, emanating from the camera and going through this seat point622. In a monocular situation this fact would establish only the rayalong which the seat point lies, and not exactly how far away this point622 is from the camera 345.

In accordance with the embodiments herein, however, the ray intersects aknown plane, and thereby defines a single point 622 in the 3-D space ofthe passenger cabin 530. Following the installation and a calibration ofthe camera, and if the seat location is known, the driver behaviormonitoring system of the example embodiment uses the full 3-Dcoordinates of the calibration seat point. With this, the driverbehavior monitoring system of the embodiment can better establish dataused for determining where in the 3-D space of the passenger cabin 530the driver's head is located.

A similar principle may be applied in accordance with a driver behaviormonitoring system of a further embodiment to find the driver's nose tip611. In this embodiment, the driver behavior monitoring system presumesthat the position of the driver's nose in the image is likely near thevertical plane 621 cutting the driver's seat in half. This preambleresults again in a line intersecting a plane and the 3-D facial normalvector origin is therefore determinable in three dimensions.

For the subject driver facing camera, the system fits a head model tothe driver's appearance, thereby obtaining a facial normal vector 522.The head model, which is generic, is rotated and scaled in 3-D spaceuntil it fits the undistorted image of the driver's head as well aspossible. The system thereby has the three angles characterizing thehead pose, to within generic head model limits, and a scale factor. Thedriver head pose angles include, for example, a driver's head pitchangle (driver looking down or up), a driver's head yaw angle (driverlooking left or right), and a driver's head roll angle (driver tiltinghis/her head to the left or right).

The system does not, however, have or otherwise know the absolutedistance 1640 (FIG. 16) of the driver from the camera, that is, thesystem does not have or otherwise know 3-D driver head locationinformation (just the angles). For this, the typical pupillary distancelimits can give the system a bound, wherein women have a mean pupillarydistance of 61.7 mm, and men have a mean pupillary distance of 64.0,both with a standard deviation of ˜3.5 mm. This renders a head distanceto within ˜±10% for ˜95% of the human population in general. That is, inthe embodiment, the system first preferentially looks for driver gender,then takes the corresponding inter-pupillary distance 1630 eye center1620 to eye center 1622 (FIG. 16) and relates the image head eye spacingto distance from the camera. Since the system has the head pose angles,the system can get the inter-pupillary distance in pixels as if thedriver were directly facing the camera. Then, using pixel size, thesystem determines the interpupillary distance in meters, apply the lensfocal length. Via similar triangles, the system calculates the head tocamera distance as:

Head to camera distance=(lens focal length*gender interpupillarydistance)/(facing the camera in the image interpupillary distance).

For instance, if there are 20 pixels separating the pupils (or eyecenters 1620, 1622, taken as proxies for the pupils), and pixels are 4microns in size, then there are 80 micrometers between the pupils. If,furthermore, the lens focal length is 2 millimeters, and driver genderis determined as male, then the camera to driver head distance is (2mm*64 mm/80 micrometers) or 1.6 meters. Given the variability in eyespacing, one may allow for this uncertainty in the final head location,and ‘soften’ the criteria for out of position warnings.

With the distance, the system is able to locate the driver's head in 3-Dspace, and then use the facial normal vector direction to relate to thevehicle cabin, mirrors, gauges, road, etc. As the facial normal vector522 typically originates at the nose tip 611, the camera to headdistance is known, and the angle to the head via the nose tip locationin the image is also known, the system of the example embodimentcalculates the facial normal vector location in space, and verifies thatthe facial normal vector “points” or is otherwise directed at or to thedesired regions around the driver, such as mirrors, road, next lane whenpassing, etc.

Overall, the driver behavior monitoring system of the embodimentmonitors the facial normal vector over time and compares the monitoredfacial normal vector with predetermined statistical properly-directedfacial normal vectors. The facial normal vector information is storedlocal in the memory of the system together with the results of thecomparison over time. These data and result may be transmitted to thecentral fleet management system as may be necessary or desired.

FIG. 13 is a flow diagram showing a method 1300 of monitoring thedriver's road attention in accordance with a combination of a locationof the driver's head and a facial normal vector of the driver's head. Animage of the cabin area of the vehicle is obtained at step 1310. A humanhead is detected in the image at step 1320. In step 1330, the locationof the human head relative to the driver-facing camera 345 and/orrelative to the various components of the cabin of the vehicle isdetermined. The facial normal vector of the detected human head isdetermined at step 1340. An estimated distance between the camera andthe driver's head is determined at step 1350. Then, at step 1360, thedriver's road attention is monitored over time using the determinedfacial normal vector of the head in combination with the determined headlocation, wherein the determined head location is used as the base pointof the facial normal vector for the monitoring.

In a further embodiment, an auto-calibration function may be realized bycollecting statistics of where the driver is looking when at highwayspeeds over time. It may be assumed that the driver is facingpredominantly forward when the vehicle is traveling over some speed,that is, the driver is very likely paying attention when moving quickly,and the most frequent or average normal vector direction will correspondto the road straight ahead. Therefore, the system of the embodimentcollects normal vector statistics by either a histogram method, arecursive average the pose angle method or a combination of thehistogram and the recursive average the pose angle methods. In thehistogram method, a histogram is created and populated for each of theset of driver's “head pose” normal vector angles describing theorientation of the driver's head, that is, a pitch histogram (driverlooking down or up), a yaw histogram (driver looking left or right), anda roll histogram (driver tilting his/her head to the left or right). Thenormal vector statistics are collected for a predetermined time, suchas, for example, 1 minute, after which the system takes the fullesthistogram bin as corresponding to a straight ahead driver's head posedirection. Alternatively, the system recursively averages the head poseangles, and determines the average value as representing straight aheaddriver's head pose direction, again letting the averages run for longenough and only when the vehicle is travelling fast enough.

Detection Of Impeded Driver-Facing Camera

Knowing that the driver-facing camera 345 in accordance with theembodiments herein vigilantly watches the drivers at all times duringoperation of the vehicle, some operators may choose to attempt to defeatthe camera for various reasons including for example, to hide fleetpolicy violations or mistakes, or the like. However, the driver facingcamera functionalities depend in large part on a clear view of thedriver. Detecting a clear view of the driver is therefore highlydesirable for the proper operation of the detection and reportingembodiments herein.

It is therefore desirable to detect whether or not a driver isattempting to defeat the driver facing camera. In this regard, impededdriver-facing camera detection systems, methods, and apparatus areprovided as described below. One benefit of these embodiments is thatproper driver facing camera operation is ensured, thereby fullysupporting the many functionalities of the several example embodimentsdescribed herein.

In accordance with an embodiment, overall, the driver facing camera 345of the driver behavior monitoring system is used to detect the driver'shead in the cabin of the vehicle during operation thereof. In theembodiment, the driver facing camera is supplemented with face detectionlogic for determining the face of the vehicle operator. The logic of theexample embodiment is executed to monitor for the continued availabilityof a visible face, of approximately unvarying appearance, when thevehicle is in motion. The logic of the example embodiment is executed togenerate a signal of a detected loss of operator verification if no faceis visible and/or determinable when the vehicle is moving.

In a further embodiment, the logic of the example embodiment includesdriver face finding functionality that executes to useforeground-background methods of object identification. The relativelystatic nature of the driver-facing camera 345 being fixedly mounted tothe vehicle headliner support member 512 (FIG. 5a ) enables theforeground-background methods of object identification for monitoringfor the continued availability of a visible driver's face, ofapproximately unvarying appearance, when the vehicle is in motion.Initially, background pixels; that is, those pixels deemed to beunchanging due to only small changes in their value, persistently covera sufficiently high percentage of the region, or even image, where aredriver's face is not expected to be seen. However, when the backgroundpixels begin to persistently cover a sufficiently high percentage of theregion, or even the image, where are driver's face may be expected to beseen, the logic of the system then determines that the image does nothave a live image of the driver and that the camera may therefore bedeemed to be impeded or otherwise blocked. If no face is visible whenthe vehicle is moving, a loss of operator verification signal isgenerated and selectively transmitted to the central fleet managementsystem or stored locally in the system memory.

In accordance with an embodiment, driver face detection logic stored ina non-transient memory device of the subject driver behavior monitoringand reporting system is executable by a processor of the system toprocess driver's head location data and a facial normal vectordetermined as described above to selectively determine, from the imagedata, a face of the driver of the associated vehicle, and generate a oneof: driver's facial characteristic data representative of theselectively determined face of the associated driver, or impeded imagedata representative of an inability of the driver face detectionlocation logic to selectively determine the face of the driver of theassociated vehicle from the image data.

The subject driver behavior monitoring and reporting system of theembodiment includes an input operatively coupled with the processor, theinput selectively receiving from the associated vehicle a vehicle movingsignal and/or a human control active signal representative of motion ofthe associated vehicle.

The control logic is executable by the processor to selectivelygenerate, responsive to the input receiving the vehicle moving signaland to the impeded image data being generated, a obstructed view datarepresentative of an obstruction between the imaging device and theassociated driver disposed in the associated vehicle.

The subject driver behavior monitoring and reporting system of theembodiment further includes driver face detection logic stored in thenon-transient memory device. The driver face detection logic isexecutable by the processor to process the image data together with thevehicle geometry data and the imaging device position data to determineone or more foreground objects in the image data and one or morebackground objects in the image data, the determined one or moreforeground objects in the image data being disposed in the associatedvehicle between the imaging device and the one or more backgroundobjects in the image data.

The driver face detection logic is further executable by the processorto process the a portion of the image data corresponding to thedetermined one or more foreground objects in the image data toselectively determine, from the image data, a face of the driver of theassociated vehicle, and generate a one of: driver's facialcharacteristic data representative of the selectively determined face ofthe associated driver, or impeded image data representative of aninability of the driver face detection location logic to selectivelydetermine the face of the driver of the associated vehicle from theimage data.

The subject driver behavior monitoring and reporting system of theembodiment further includes an input operatively coupled with theprocessor, the input selectively receiving from the associated vehicle avehicle moving signal representative of motion of the associatedvehicle. In the embodiment, the control logic is executable by theprocessor to selectively generate, responsive to the input receiving thevehicle moving signal and to the impeded image data being generated, aobstructed view data representative of an obstruction between theimaging device and the associated driver disposed in the associatedvehicle.

FIG. 14 is a flow diagram showing a method 1400 of monitoring thepresence of a driver's face in accordance with an example embodiment.The time is determined at step 1402 and an image of the vehicle cabin isobtained in step 1404. The time may be associated with the cabin imagedata as necessary or desired. The cabin image data is searched at step1406 to find a human face, in the approximate location where thedriver's face may be expected to be.

A determination is made at step 1410 whether a driver's face is found inthe cabin image in step 1406. If no face is found a furtherdetermination is made at step 1412 whether the vehicle is moving. If noface is found and the vehicle is moving, a warning signal is generatedat step 1414, and the warning signal is selectively transmitted at step1416 to the central fleet management system. Alternatively, the warningsignal may be stored locally in the memory of the driver behaviormonitoring system of the embodiment.

Driver's Head Out Of Position

Many vehicle operators reach for items while driving such as, forexample, control knobs on the dashboard, cups stowed in nearby cupholders, maps or other items stowed in a center console or door pocketnext to the driver's seat, or the like. This is of course normalbehavior. However, it has been found that reaching to gain access tofaraway objects while driving increases the chances of an accident by afactor of about eight (8).

It is therefore desirable to measure and warn for an out of normalposition head, as this correlates to excessively reaching. The driver'shead position is used in the example embodiment as a proxy for thedriver's reach and, in particular, the driver's head position is used inthe example embodiment as a proxy for the driver's excessive reachingthereby generating a signal representative of this monitored driverbehavior.

The example embodiment to be described herein provides a verification ofthe driver not excessively reaching for items beyond his considered tobe safe grasp space, preferably an extent of a reach maneuver capable ofbeing performed by the driver without excessive body movement.Understanding of typical driver head position and warning when thedriver over-reaches in accordance with the example embodiments isbeneficial to help prevent accidents caused by driver inattention.

The driver behavior monitoring system of the example embodiment uses thedriver facing camera 345 to locate and measure the driver's headposition. Logic executing in the driver behavior monitoring system usesrecursive measurement equations to determine the mean and variance ofthe set of determined driver's head positions. The logic executinggenerates a warning or notice signal when a driver's head positiondeviates from the mean position by more than a predetermined number ofstandard deviations in any axis (x−y−or z−) and when this deviationoccurs for a predetermined minimum time period. The predetermined numberof standard deviations and the predetermined minimum time to be out ofposition are parameters that are settable or otherwise selectable by theoperator or fleet system manager. Typical values of these settableparameters may be two (2) standard deviations, essentially coveringabout 95% of a normally distributed variable, and for approximately 1-2seconds. The driver's head out of position events are determined andrecorded into the local memory device of the driver behavior monitoringsystem of the example embodiment. The driver's out of position behavioris recorded by the camera 345 and may be stored together with other datarelating to the operation of the vehicle at the time the driver's headwas out of position such as, for example, vehicle speed data or thelike. An out of head position indication combined with a high vehiclespeed indication from the vehicle speed sensors may be used by thesystem to grade or otherwise score the head out of position occurrencemore negatively than for example an out of head position indicationcombined with a very low vehicle speed indication from the vehicle speedsensors. Stopping the vehicle to reach for items beyond the driver'sconsidered to be safe grasp space is graded or otherwise scored by thedriver behavior monitoring system of example embodiment to be gooddriver behavior. Conversely, continued operation of the vehicle athighway speeds for example while reaching for items beyond the driver'sconsidered to be safe grasp space is graded or otherwise scored by thedriver behavior monitoring system of example embodiment to be bad driverbehavior. Other one or more vehicle conditions may be monitored andcombined with the driver's head position used in the example embodimentsas a proxy for the driver's reach for determining a level of driverbehavior on a good to bad scale.

FIG. 15 is a flow diagram showing a method 1500 of monitoring theposition of the driver's head used as a proxy for the driver's reachand, in particular, used in as a proxy for the driver's excessivereaching, in accordance with an example embodiment. The time isdetermined at step 1502 and an image of the vehicle cabin is obtained instep 1504. The time may be associated with the cabin image data asnecessary or desired. The cabin image data is searched at step 1506 tofind a human head, preferably the driver's head. The location of thedriver's head is stored into a local memory in order that, in step 1508,the mean and variance of the driver's head position may be determinedover a predetermined time interval.

A determination is made at step 1510 whether the driver's head positionis outside of the mean and/or variance values determined in step 1508.In an embodiment, the determination made at step 1510 of whether thedriver's head position is outside of the mean and/or variance valuesdetermined in step 1508 includes determining whether the driver's headposition is outside of the mean and/or variance values for apredetermined time period, which may be selectable by the operator orfleet manager. A head position warming signal is generated at step 1530indicating that the driver's head position is outside of the mean and/orvariance values for a predetermined time period. A video image of thedriver is recorded at step 1532, and the head position warming signaland the video image of the driver are selectively transmitted to thecentral fleet manager in step 1534.

Alternatively, the head position warming signal and the video image ofthe driver may be stored locally in the memory of the driver behaviormonitoring system of the embodiment.

FIG. 15a is a flow diagram showing a method 1550 of determining whetherthe driver's head is out of position in accordance with an exampleembodiment, with a particular focus on collecting statistics of a“normal” driver's head position such as, for example while the vehicleis moving sufficiently fast enough and for a sufficiently long enoughperiod, prior to assessing the driver's head out of position inaccordance with the collected statistics. A timer is initialized in step1560, and the driver's head pose statistics are collected at step 1562.Preferably, the driver's head pose statistics are collected when thevehicle is moving quickly enough, and for long enough. The driver's headpose mean and variance values need in the example embodiment, some timeto develop before they have any practical value such as, for example, onthe scale of about one (1) minute at speed. Only after driver's headpose mean and variance values are collected and developed at step 1562does the system of the embodiment know what is ‘regular’ driving forthis driver, and only then does the system perform driver's head out ofposition testing. This test first consists of imaging the driver toobtain at step 1564 a current driver's image. A comparison is performedat step 1570 between the current measured head pose values (yaw, pitch,roll, location) and the mean values of these driver head pose anglesincluding for example a driver's head pitch (driver looking down or up),a driver's head yaw (driver looking left or right), and a driver's headroll (driver tilting his/her head to the left or right) developed atstep 1562. If any of these deviates by more than a selectable amount ofstandard deviations, preferably about two (2) standard deviations fromthe corresponding mean, the system deems the driver's head to be out ofposition. A timer is started in step 1572 when the head is out ofposition. Should the value of the timer exceed a threshold as determinedat step 1574, a warning is issued at step 1580. When the head is not outof position, the timer is reset to zero at step 1582.

In accordance with the example embodiment, control logic of the driverbehavior monitoring and reporting system is executable by a processor ofthe system to determine, over a predetermined detection time, a centralvalue of a facial normal vector of the driver of a vehicle, and todetermine, over the predetermined detection time, a dispersion of thecentral value of the facial normal vector. The mean of the head positionvalue of the facial normal vector may be determined and a variance ofthe facial normal vector may be determined to render a standarddeviation of the driver's head position as the square root of thevariance.

A memory device of the stores, as the driver road attention parameter ofthe safe attention model data, a recommended value range of a driverhead out of position parameter of the monitored driver attentioncondition as a selectable multiple of the determined standard deviationof the facial normal vector.

The control logic stored in the non-transient memory device isexecutable by the processor of the driver behavior monitoring andreporting system to process the facial normal vector to determine anoperational value of the driver road attention parameter of themonitored driver attention condition of the associated vehicle, and toperform a comparison between the recommended value range of the driverhead out of position parameter of the monitored driver attentioncondition of the associated vehicle and the determined operational valueof the driver head out of position parameter of the monitored driverattention condition of the associated vehicle.

The control logic stored in the non-transient memory device is furtherexecutable by the processor of the driver behavior monitoring andreporting system to determine a driver inattention value as a driverinattention value.

The control logic may further determine the state of vehicle operationcompliance in a binary sense as a one of a driver inattention state inaccordance with a first result of the comparison between the recommendedvalue range and the determined operational value of the driver head outof position parameter of the monitored driver attention condition of theassociated vehicle, wherein the processor generates the driverinattention data in accordance with the first result, or a driverattention state in accordance with a second result of the comparisonbetween the recommended value range and the determined operational valueof the driver head out of position parameter of the monitored driverattention condition of the associated vehicle.

Driver's Head Pose Distribution Metric

One aspect of good driving behavior may be characterized as the driverbeing in their proper, individual, driving position, i.e. able to holdthe steering wheel, able to see forward to the roadway, able to see themirrors, positioned within reach of the pedals, and the like.Essentially, good body position within the vehicle will usually lead toan optimized driver performance. Deviations from these operationalpositions are associated with a greater risk of accidents, by up to afactor of about eight (8) as noted above. Another aspect of good drivingbehavior may be characterized as the driver actually looking where theyshould when they drive. For instance, mirrors shall be utilized whenbacking, so eyes off the forward road under these conditions isacceptable, and eyes on one of the vehicle mirrors is desired. Lighttraffic while moving forward might require the driver to scan theforward road often with periodic mirror scans, but with most attentionbeing paid to the forward road. However, dense traffic situationsprobably require more scanning of the side mirrors than with littletraffic. Lane changes are beneficially prefaced by looking at the laneinto which one is going.

It is desired therefore to detect improper or deviant head directionbehavior, particularly against the background of the current drivingmaneuver, and to use this as a monitored behavior event. The driver maybe warned by the system when an improper or deviant behavior occurs. Asignal may be generated when the improper or deviant behavior occurs anddata representative of the signal may be stored locally in the vehiclemounted monitoring system or transmitted to the central fleet managementsystem. Still images or video images of the cabin of the vehicle may berecorded when the improper or deviant behavior occurs and datarepresentative of the cabin images taken during the improper or deviantbehavior may be stored locally in the vehicle mounted monitoring systemor transmitted to the central fleet management system. In an embodiment,the resolution/compression quality of the driver behavior recorded bythe driver facing camera may be adjusted during the improper or deviantbehavior to improve or otherwise enhance the video quality to reflectthat this is a head pose driver behavior event.

The driver behavior monitoring system of the embodiment determines adriver's head pose using the driver-facing camera, logic and a processorexecuting the logic, determines a distribution of the head pose overtime, and monitors the distribution of the head pose, for warning thedriver when this deviates from a desired or usual distribution. Awarning signal may be generated and/or a warning event may be triggeredfor storing data related to the warning signal indicating the head posedeviating from the desired or usual distribution. The warning signaland/or the data related to the warning signal may be transmitted to thecentral fleet management system.

Overall, the system observes the driver's head pose (facing direction)using the driver facing camera 345. The spatial distribution of thedriver's head pose is collected over time, and generate a 3-D histogramof head roll, pitch and yaw is generated. The driver behavior monitoringsystem is then able to verify that there is a (desired and proper)change in the histogram when the driver is engaged in a vehicle backingactivity, when engaged in a turning (look left when turning left, forinstance) activity, and when performing other actions with the vehicle.By means of change detection methods, significant deviations from thedriver's normal pose distribution may be detected from the head posedata collected, and the detected deviations may be flagged such as forexample by generating a driver head pose deviation signal.

In an embodiment, the histogram is operable on two time scales. That is,the histogram is operable on a long time scale, for learning orotherwise developing the driver's ‘average’ behavior, and the histogramis operable on a short time scale, for learning or otherwise developingthe driver's ‘what is happening now’ driver behavior. The two histogramsare compared in the embodiment.

FIG. 16 is a diagrammatic showing an image 1600 (not taken by the driverfacing camera of the embodiments) of a cabin 1610 of an associatedvehicle illustrating the driver facing camera 345 in accordance with theembodiment imaging a properly seated driver 1612 appropriately lookingat the vehicle mirror 1650. The driver behavior monitoring system andfits a head pose model shown in the drawing Figure as a driver's headpose vector 1660 originating at the driver's nose 1624. This vector 1660may be visualized as a rigidly affixed handle connected to a generic 3-Dface model. The face model is tilted, turned, and adjusted angularly andscale-wise until it fits the observed face as closely as possible. The3-D angles corresponding to the handle are the head pose. It is to beappreciated that the head pose model embraces and otherwise includesdriver head location information, driver head roll information, anddriver head pitch and yaw information.

As described above, for the subject driver facing camera 345, the systemfits a head model to the driver's appearance, thereby obtaining a facialnormal vector 1660. The head model, which is generic, is rotated andscaled in 3-D space until it fits the undistorted image of the driver'shead as well as possible. The system thereby has the three anglescharacterizing the head pose, to within generic head model limits, and ascale factor. The driver head pose angles include, for example, adriver's head pitch angle (driver looking down or up), a driver's headyaw angle (driver looking left or right), and a driver's head roll angle(driver tilting his/her head to the left or right).

The system does not, however, have or otherwise know the absolutedistance 1640 (FIG. 16) from the camera 345 to the driver 1612, that is,the system does not have or otherwise know 3-D driver head locationinformation (just the angles). The typical pupillary distance 1630limits can give the system a bound, wherein women have a mean pupillarydistance of 61.7 mm, and men have a mean pupillary distance of 64.0,both with a standard deviation of ˜3.5 mm. This renders a head distanceto within ˜±10% for ˜95% of the human population in general. That is, inthe embodiment, the system first preferentially looks for driver gender,then takes the corresponding inter-pupillary distance 1630 eye center1620 to eye center 1622 and relates the image head eye spacing todistance from the camera. Since the system has the head pose angles, thesystem can determine or otherwise calculate the inter-pupillary distance1630 in pixels as if the driver 1612 were directly facing the camera345. Then, using pixel size, the system determines the interpupillarydistance 1630 in meters, apply the lens focal length. Via similartriangles, the system calculates the distance between the camera 345 andthe driver's 1612 head as:

Head to camera distance=(lens focal length*gender interpupillarydistance)/(facing the camera in the image interpupillary distance).

For instance, if there are 20 pixels separating the pupils (or eyecenters 1620, 1622, taken as proxies for the pupils), and pixels are 4microns in size, then there are 80 micrometers between the pupils. If,furthermore, the lens focal length is 2 millimeters, and driver genderis determined as male, then the camera to driver head distance is (2mm*64 mm/80 micrometers) or 1.6 meters.

With the distance, the system is able to locate the driver's head in 3-Dspace, and then use the facial normal vector 1660 direction to relate tothe vehicle cabin, mirrors, gauges, road, etc. As the facial normalvector 1660 typically originates at the nose tip 1624, the camera tohead distance is known, and the angle to the head via the nose tiplocation in the image is also known, the system of the exampleembodiment calculates the facial normal vector location in space, andverifies that the facial normal vector “points” or is otherwise directedat or to the desired regions around the driver, such as mirrors, road,next lane when passing, etc.

The system may collect data over a selectable period of time such as,for example, over the last 120 seconds of the driver's head pose,entering this collected data into a multi-dimensional histogram storedin the local memory of the system. It is preferred that a circular listsupplemented with a pointer to the oldest entry computational structuremay form the data storage backbone feeding this histogram.

The histogram may then be compared with an observed safe condition. Theobserved safe condition may possibly be derived from the statistics ofone or more accident-free time histories, or from one or morepredetermined set of statistics of accident-free time history models.Still further, the histogram may be compared with a desired histogram ofthe fleet associated with the vehicle. Examples of comparing histogramsare disclosed, for example, in Serratosa F., Sanroma G., Sanfeliu A.(2007) “A New Algorithm to Compute the Distance BetweenMulti-dimensional Histograms” In: Rueda L., Mery D., Kittler J. (eds)Progress in Pattern Recognition, Image Analysis and Applications. CIARP2007. Lecture Notes in Computer Science, vol 4756. Springer, Berlin,Heidelberg, the teachings of which are incorporated herein by reference.

FIG. 17 is a flow diagram showing a method of operating a driverbehavior monitoring system having a driver facing camera for detecting,monitoring, and reporting whether the driver's head pose distribution issignificantly changing or unacceptable implementing a driver roadattention strategy in accordance with an example embodiment. Withreference now to that Figure, in the method 1700 of the embodiment, thedriver image data collection portion 832′ includes a step 1702determining a time of the image of the driver, and a step 1704collecting the image of the driver. In step 1106 the logic of the systemdetermines information relating to the operation of the vehicle such as,for example, vehicle speed data or the like, and the logic alsodetermines the head pose of the driver. The historical driver's headpose data is updated in step 1708 with the newly acquired driver's headpose.

A determination is made in step 1710 whether the collected historicaldata differs from a predetermined desired distribution for a givenvehicle state. If the collected historical data does not differ from thepredetermined desired distribution for the given vehicle state, noaction is taken. However, if the collected historical data does differfrom the predetermined desired distribution for the given vehicle state,then the method 1700 generates at step 1730 a head pose warning signaland/or generates head pose warning data. A video image of the driver isrecorded or otherwise collected at step 1732, and the head pose warningsignal and/or the head pose warning data is selectively transmitted instep 1734 together with the video image of the driver to a central fleetmanagement system or the like. Alternatively, the video image of thedriver and the head pose warning signal and/or the head pose warningdata may be selectively stored in a memory device of the drivermonitoring system local to the vehicle.

FIG. 18 is an example of a head pose distribution map 1800 in accordancewith an example embodiment. As illustrated in that Figure, avisualization and analysis framework of the head pose distribution maybe performed in spherical coordinates, mapping to named locations. Themapped locations may include, for example, a location of the vehicleradio 1822, a location of the right and left footwells of the vehicle1824 and 1826, a location of the driver's lap 1825, a location of apassenger in the vehicle 1828, a location of the left and right mirrorsof the vehicle 1830 and 1832, a location of the sunvisor of the vehicle1850, or a location of the roadway straight ahead 1850. A color tinted“heat” map (i.e. histogram) may indicate the frequency with which eachlocation is faced is illustrated in that Figure wherein the heat maphaving the highest driver focus intensity is sketched with “x” markersfor the presumably often viewed forward roadway ahead of the vehicle.Portions of the map may be associated to labels—for instance, when theradio station is being changed and the driver is not facing forward inthe normal pose, and somewhat to the right, then the map area beingfaced may be labeled radio (or the likelihood of it being the radioincreases). Similar labeling schemes may be used for the mirrors, thistime triggered by a set blinker turn signal, and the driver turning leftor right, in the sense of the turn signal.

It should be observed that the safe driving position may vary,temporarily or longer term. For instance, the user may need to adjust acontrol that is further away (e.g. a fan, perhaps) or the user maychange the seat position (e.g. to relieve a sore back). We may thereforeneed to perform a histogram restart or mask out measurement values whenthese, perhaps temporary, perhaps persistent, changes occur.

FIG. 19 is a basic flow diagram showing a method 1900 of comparingdriver head pose histograms, and determining and reporting deviationsand/or changes between the driver head pose histograms in accordancewith an embodiment. Turning now to that Figure, the method 1900determines improper or deviant driver head direction behavior based on adriver's head pose distribution metric. The method 1900 includes a startstep 1910 which, thereafter, initiates a step 1912 of the system imagingthe driver and cabin of the associated vehicle and obtaining driverimage data. The driver's head pose is measured in step 1914, and adriver's head pose histogram of the last n seconds of driver head imagecapturing is created in step 1916.

Next, in step 1920 the system determines whether the histogram shows adifference between the desired driver behavior and the actual driverbehavior. If there is no difference between the desired driver behaviorand the actual driver behavior, or if the difference is within apredetermined bounds, the system repeats step 1912 whereupon the systemagain images the driver and cabin of the associated vehicle and obtainsnew driver image data. On the other hand, if there is a differencebetween the desired driver behavior and the actual driver behavior, orif the difference is outside of the predetermined bounds, the systeminitiates step 1922 whereupon the system generates the driverinattention signal as determined based on the driver's head posedistribution metric.

FIG. 19a is a flow diagram showing a method 1950 of determining whetherthe driver's head is out of position in accordance with an exampleembodiment, with a particular focus on collecting statistics of a“normal” driver's head pose such as, for example while the vehicle ismoving sufficiently fast enough and for a sufficiently long enoughperiod, prior to assessing the driver's head pose in accordance with thecollected statistics. A timer is initialized in step 1960, and thedriver's head pose statistics are collected at step 1962. Preferably,the driver's head pose statistics are collected when the vehicle ismoving quickly enough, and for long enough. The driver's head pose meanand variance values need in the example embodiment, some time to developbefore they have any practical value such as, for example, on the scaleof about one (1) minute at speed. Only after driver's head pose mean andvariance values are collected and developed at step 1962 does the systemof the embodiment know what is ‘regular’ driving for this driver, andonly then does the system perform driver's head pose testing. This testconsists of imaging the driver to obtain at step 1964 a current driver'simage. A comparison is performed at step 1970 between the currentmeasured head pose values (yaw, pitch, roll, location) and the meanvalues of these driver head pose angles including for example a driver'shead pitch (driver looking down or up), a driver's head yaw (driverlooking left or right), and a driver's head roll (driver tilting his/herhead to the left or right) developed at step 1962. If any of thesedeviates by more than a selectable amount of standard deviations,preferably about two (2) standard deviations from the correspondingmean, the system deems the driver's head to be out of position. A timeris started in step 1972 when the head is out of position. Should thevalue of the timer exceed a threshold as determined at step 1974, awarning is issued at step 1980. When the head is not out of position,the timer is reset to zero at step 1982.

FIG. 20 is a flow diagram showing a method 2000 of comparing head posedistribution maps, and determining and reporting deviations between theactual map and a desired, situation appropriate, map in accordance withan example embodiment. The embodiment has a particular focus oncollecting statistics of a “normal” driver's head position such as, forexample while the vehicle is moving sufficiently fast enough and for asufficiently long enough period, prior to assessing the driver's headout of position in accordance with the collected statistics. A timer isinitialized in step 2060, and the driver's head pose statistics arecollected at step 2062. Preferably, the driver's head pose statisticsare collected when the vehicle is moving quickly enough, and for longenough. The driver's head pose mean and variance values need in theexample embodiment, some time to stabilize before they have anypractical value such as, for example, on the scale of about one (1)minute at speed. Only after driver's head pose mean and variance valuesare collected and developed at step 2062 does the system of theembodiment know what is ‘regular’ driving for this driver, and only thendoes the system perform driver's head out of position testing. This testfirst consists of imaging the driver to obtain at step 2064 a currentdriver's image. A comparison is performed at step 2070 between thecurrent measured head pose values (yaw, pitch, roll, location) and ahistogram of driver head pose angles including for example a driver'shead pitch (driver looking down or up), a driver's head yaw (driverlooking left or right), and a driver's head roll (driver tilting his/herhead to the left or right) developed at step 2062. If any of thesedeviates by more than a selectable amount of standard deviations,preferably about two (2) standard deviations from the correspondingmean, the system deems the driver's head to be out of position. A timeris incremented in step 2072 when the head is out of position. Should thevalue of the timer exceed a threshold as determined at step 2074, awarning is issued at step 2080. When the head is not out of position,the timer is reset to zero at step 2082.

Driver's Eyes On Road With Adaptive LDW Warning Margin

Drivers not properly looking at the road when driving forward willlikely need a longer time to react to a dangerous situation. It istherefore desirable to adjust the warning parameters for a dangerdetection system, such as a lane departure warning device or aradar-based distance keeping aid, such that the driver is warned in amore timely fashion.

The system of the example embodiment therefore couples the time thedriver is not looking at the road ahead with an increased warning marginparameter. A linear relationship may be used for instance, such as:

Warning parameter=base warning parameter value+(factor*(elapsed timesince driver has last looked at road)).

In the example embodiment, the resulting warning parameter value is thencapped at some maximum value and/or number, which may be selectable bythe driver, a fleet manager, or the like. The elapsed time since thedriver has last looked at the road may have, in accordance with afurther embodiment, a ‘grace period’ value subtracted before it is usedin the above equation. This beneficially allows the driver to brieflyglance away, during which time the vehicle warning systems do not changetheir parametrization. It is understood that an equivalent negativevalue version or an adjustment in a decreasing magnitude sense for theabove equation may also apply, as required by the application using theparameter.

The factor in the above equation may be adjusted within limits so that adesired driver behavior is maintained, e.g. so that the headway timestays greater than some minimum value for at least 95% of the time. Thisadjustment may be made by the driver or from a fleet command center,which can observe the driver's safety relevant behavioral statistics.

Driver's Mirror Usage Verification

Commercial vehicle drivers have many tasks to coordinate during vehicleoperation. One of these tasks is scanning the vehicle mirrors. When thevehicle mirror scanning is not done properly or is not done withsufficient frequently, collision risk increases.

It is desirable, therefore to provide a system, method and apparatus forverifying the sufficiency and adequacy of the drover's mirror usage. Inaccordance with an embodiment, the driver facing camera 345 is used toverify the driver's proper use of the mirrors of the vehicle.

The embodiments advantageously provide improvements in vehicle operationby helping to increase driving safety, both for commercial and othervehicles as well as for other vehicles around the vehicle having thedriver behavior monitoring systems, methods and apparatus of theembodiments herein including in particular the embodiment providingmirror usage verification. The embodiment further providecharacterization of the driver such as, for example, biometric IDinformation, and warn the driver and remote fleet management if anyunsafe behavior occurs or is detected.

Algorithms for finding faces in images use a model of the human face.This model typically looks for facial ‘landmarks’, that is, contrasting,distinct, areas, such as the corners of the mouth, eyes, etc. When aconfiguration of such landmarks is found that is within the geometricexpectations for human facial appearance, the face is located.

The configuration of the landmarks relates to the direction in which theface points (its ‘pose’) relative to the camera. The pose may besummarized by a 3-dimensional vector originating at the person's nose asshown in FIGS. 5a and 17 as a 3-D head pose vector 522.

It may also be seen that the face has been located (chin, mouth, eyes,etc), placing it within a certain volume in the passenger cabin. The tipof the nose is located on a ray emanating from the camera, and onaverage approximately centered on the seat and pointing straightforward.

FIG. 21 is an illustration of the bounds applying to mirror usage inaccordance with an example embodiment. The system of the embodimentrelates the facial pose vector 522, together with the head position, tosee in what direction the driver is facing (not necessarily the same aslooking, or gazing). Though glancing by eye motion only at the mirrorsis possible, the system examines the facial pose vector 522 over time todetermine whether the driver is moving their head to look—as theyshould—at the mirrors. When the driver is not looking at the mirrorsoften enough 2120— or perhaps for too long 2110 (after all, one shouldmostly look forward when driving forward, for example), a warning isissued, and a Safety Event Recording may be triggered, and statisticsregarding driver behavior may be collected.

The system of the embodiment can thus use the driver facing camera 345(whose position and geometry is known, together with the driver's headlocation and pose, to increase safety, enforce policy, look for hints offatigue, and collect safety and driver behavior statistics.

A particular case of mirror usage verification is that of changinglanes. Good driving practice states that the mirror associated with thelane that one is changing into shall be used before the lane change ismade. Therefore, when the turn signal is set, for example, the system ofthe example embodiment executes a test for using the mirror before thelane change. The test may be, for example, to determine whether thedriver looked at the appropriate mirror for long enough (between theupper 2110 and lower 2120 bands) before the lane change. Equivalently,if the turn signal is not set, but the lane is changed (an eventdetectable by a lane departure warning system), and the mirror is notlooked at, then this ‘not using the mirror before a lane change isdetected’ event is also triggered.

A similar test for mirror usage may be performed when a driver isstanding still and blinking to the right. This is a classic, dangerous,situation for any cyclists located on a commercial vehicle's right side,where they may be crushed by the turning truck. One may thereforeenforce proper mirror usage by verifying that the driver has looked tothe right before the vehicle moves again, that is, create a visualinterlock on vehicle movement. It is understood that the left sideversion of this may also be similarly implemented in regions whereleft-side traffic is the norm.

In accordance with an embodiment, a system monitoring a driver attentioncondition of an associated vehicle during operation of the associatedvehicle by an associated driver is provided. The system includes animaging device disposed in the associated vehicle, a control deviceincluding a processor, and an output operatively coupled with theprocessor. The imaging device captures an image of the associated driverdisposed in the associated vehicle and of an interior of the associatedvehicle, and generates image data representative of the captured imageof the associated driver disposed in the associated vehicle and of theinterior of the associated vehicle,

The control device includes an image data input operatively coupled withthe processor, a non-transient memory device operatively coupled withthe processor, driver head detection logic stored in the non-transientmemory device, driver head direction logic stored in the non-transientmemory device, and control logic stored in the non-transient memorydevice.

The image data input receives the image data from the imaging device.The non-transient memory device stores vehicle geometry datarepresentative of relative positions between one or more structures ofthe associated vehicle, imaging device position data representative of aposition of the imaging relative to the one or more structures of theassociated vehicle, and safe attention model data comprising arecommended value range of a driver road attention parameter of themonitored driver attention condition of the associated vehicle.

The driver head detection logic is executable by the processor toprocess the image data to locate/determine a head candidate area of theimage captured by the imaging device likely above a predeterminedthreshold stored in the non-transient memory device to be representativeof the head of the associated driver disposed in the associated vehicle,and tag a portion of the image data corresponding to the head candidatearea located/determined by the driver head detection logic as driverhead image data.

The driver head direction logic is executable by the processor toprocess the driver head image data to determine a facing direction ofthe head of the associated driver, and generate driver head facingdirection data, the driver head facing direction data beingrepresentative of the determined facing direction of the head of theassociated driver.

The control logic is executable by the processor to process the driverhead facing direction data together with the vehicle geometry data andthe imaging device position data to determine an operational value ofthe driver road attention parameter of the monitored driver attentioncondition of the associated vehicle, and perform a comparison betweenthe recommended value range of the driver road attention parameter ofthe monitored driver attention condition of the associated vehicle andthe determined operational value of the driver road attention parameterof the monitored driver attention condition of the associated vehicle.

The control logic is further executable by the processor to determine astate of vehicle operation compliance in accordance with a result of thecomparison between the recommended value range and the determinedoperational value of the driver road attention parameter of themonitored driver attention condition of the associated vehicle.

The control logic may in accordance with an example determine the stateof the vehicle operation compliance as a one of a driver inattentionstate in accordance with a first result of the comparison between therecommended value range and the determined operational value of thedriver road attention parameter of the monitored driver attentioncondition of the associated vehicle, wherein the processor generatesdriver inattention data in accordance with the first result, or a driverattention state in accordance with a second result of the comparisonbetween the recommended value range and the determined operational valueof the driver road attention parameter of the monitored driver attentioncondition of the associated vehicle.

The an output selectively receives the driver inattention data from theprocessor and generates a driver inattention signal representative ofthe determined operational value of the driver road attention parameterof the monitored driver attention condition being outside of therecommended value range of the safe model data.

In accordance with a further example embodiment, the control logic isexecutable by the processor to process driver head facing direction datatogether with vehicle geometry data and imaging device position data todetermine an operational value of the driver road attention parameter ofthe monitored driver attention condition of the associated vehicle,correlate the driver road attention parameter of the monitored driverattention condition of the associated vehicle with an operational valueof a parameter of a lane departure warning (LDW) monitored condition ofthe associated vehicle, and determine an adjustment value for modifyingsetting a LDW system of the associated vehicle in accordance with thedriver road attention parameter of the monitored driver attentioncondition of the associated vehicle correlated with the operationalvalue of the parameter of the LDW monitored condition of the associatedvehicle. The output is operatively coupled with an input of the LDWsystem of the associated vehicle, and selectively receives theadjustment value for modifying the LDW setting, and delivers theadjustment value to the associated vehicle.

It is now claimed:
 1. A safety system, comprising: an imaging devicedisposed in an associated vehicle, the imaging device capturing an imageof an associated driver disposed in the associated vehicle and of aninterior of the associated vehicle, and generating driver head imagedata representative of the captured image of the associated driverdisposed in the associated vehicle and of the interior of the associatedvehicle; and a control device comprising: a processor; an image datainput operatively coupled with the processor, the image data inputreceiving the driver head image data from the imaging device; anon-transient memory device operatively coupled with the processor, thenon-transient memory device storing safe attention model data comprisinga recommended value of a driver road attention parameter of a monitoreddriver attention condition of the associated vehicle and at least one ofi) recursively measured statistical values based on a sufficient numberof measurements derived at a sufficient speed of the associated vehicleand ii) a statistical value derived from a histogram associated with astraight ahead driver's head pose direction derived at the sufficientspeed of the associated vehicle; driver head direction logic stored inthe non-transient memory device, the driver head direction logic beingexecutable by the processor to: process the driver head image data todetermine a facing direction of the head of the associated driver; andgenerate driver head facing direction data, the driver head facingdirection data being representative of the determined facing directionof the head of the associated driver; control logic stored in thenon-transient memory device, the control logic being executable by theprocessor to: process the driver head facing direction data to determinean operational value of the driver road attention parameter of themonitored driver attention condition of the associated vehicle; performa comparison between the recommended value of the driver road attentionparameter and the determined operational value of the driver roadattention parameter of the monitored driver attention condition of theassociated vehicle; determine driver road attention compliance inaccordance with a result of the comparison between the recommended valueand the determined operational value of the driver road attentionparameter of the monitored driver attention condition of the associatedvehicle; relate the determined driver road attention compliance to anoperational value of a parameter of a monitored condition of a safetyevent system; determine an adjustment value for modifying the parameter,wherein the adjustment value is determined according to a functionalrelationship based on the operational value and a factor, a desireddriver behavior being maintained by the functional relationship; andtransmit the adjustment value for modifying the setting of the safetyevent system.
 2. The safety system as set forth in claim 1, wherein thesafety event system is on the associated vehicle, the safety systemfurther including: an output operatively coupled with the processor andwith an input of the safety event system, the output selectivelyreceiving the transmitted adjustment value for modifying the safetyevent system setting, and delivering the adjustment value to the safetyevent system for effecting a modification of the setting of the safetyevent system.
 3. The safety system as set forth in claim 1, wherein: themonitored driver attention condition is based on an elapsed time thedriver head facing direction data indicates the associated driver lastlooked at a roadway along which the associated vehicle is traveling. 4.The safety system as set forth in claim 1, wherein: the safety eventsystem is a lane departure warning system.
 5. The safety system as setforth in claim 1, wherein: the safety event system is danger detectionsystem.
 6. The safety system as set forth in claim 1, wherein: thenon-transient memory device stores position data representative of aposition of the imaging device relative to the one or more structures ofthe associated vehicle; and the driver head location logic is executableby the processor to: process the driver head image data together withthe imaging device position data to determine a location of the driver'shead relative to the one or more structures of the associated vehicle;and generate driver's head location data, the driver's head locationdata being representative of the determined location of the head of theassociated driver relative to the one or more structures of theassociated vehicle.
 7. The safety system as set forth in claim 6,wherein: the one or more structures of the associated vehicle includes awindshield.
 8. The safety system as set forth in claim 6, wherein: thecontrol logic is executable by the processor to process the driver headfacing direction data together with the driver's head location data todetermine the operational value of the driver road attention parameter.9. The safety system as set forth in claim 8, wherein: if the determineddriver road attention compliance indicates the associated driver is inan inattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time basedon the monitored condition.
 10. The safety system as set forth in claim9, wherein: the safety event system is a lane departure warning system;the monitored condition of the lane departure warning system is theassociated vehicle crossing a lane; if the determined driver roadattention compliance indicates the associated driver is in aninattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time if theassociated vehicle crosses the lane.
 11. The safety system as set forthin claim 9, wherein: the safety event system is headway keeping aid. 12.The safety system as set forth in claim 11, wherein: the headway keepingaid is a headway distance keeping aid; the monitored condition of theheadway distance keeping aid is a distance to a forward vehicle; and ifthe determined driver road attention compliance indicates the associateddriver is in an inattention state, the operational value of theparameter of the monitored condition of the safety event system ismodified by the adjustment value to warn the associated driver at anearlier time if the distance from the associated vehicle to the forwardvehicle is less than a predetermined headway distance.
 13. The safetysystem as set forth in claim 11, wherein: the headway keeping aid is aheadway time keeping aid; the monitored condition of the headway timekeeping aid is a time to a forward vehicle; and if the determined driverroad attention compliance indicates the associated driver is in aninattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time if thetime from the associated vehicle to the forward vehicle is less than apredetermined headway time.
 14. The safety system as set forth in claim9, wherein: the safety event system is a collision mitigation brakingsystem; the monitored condition of the collision mitigation brakingsystem is a collision mitigation braking event; if the determined driverroad attention compliance indicates the associated driver is in aninattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time of thecollision mitigation braking event.
 15. A safety system, comprising: animaging device disposed in an associated vehicle, the imaging devicecapturing an image of an associated driver disposed in the associatedvehicle and of an interior of the associated vehicle, and generatingdriver head image data representative of the captured image of theassociated driver disposed in the associated vehicle and of the interiorof the associated vehicle; a control device comprising: a processor; animage data input operatively coupled with the processor, the image datainput receiving the driver head image data from the imaging device; anon-transient memory device operatively coupled with the processor, thenon-transient memory device storing safe attention model data comprisinga recommended value of a driver road attention parameter of a monitoreddriver attention condition of the associated vehicle and at least one ofi) recursively measured statistical values based on a sufficient numberof measurements derived at a sufficient speed of the associated vehicleand ii) a statistical value derived from a histogram associated with astraight ahead driver's head pose direction derived at the sufficientspeed of the associated vehicle; driver head direction logic stored inthe non-transient memory device, the driver head direction logic beingexecutable by the processor to: process the driver head image data todetermine a facing direction of the head of the associated driver; andgenerate driver head facing direction data, the driver head facingdirection data being representative of the determined facing directionof the head of the associated driver; control logic stored in thenon-transient memory device, the control logic being executable by theprocessor to: process the driver head facing direction data to determinean operational value of the driver road attention parameter of themonitored driver attention condition of the associated vehicle; performa comparison between the recommended value range of the driver roadattention parameter and the determined operational value of the driverroad attention parameter of the monitored driver attention condition ofthe associated vehicle; determine driver road attention compliance inaccordance with a result of the comparison between the recommended valuerange and the determined operational value of the driver road attentionparameter of the monitored driver attention condition of the associatedvehicle; relate the determined driver road attention compliance to anoperational value of a parameter of a monitored condition of a safetyevent system of the associated vehicle; determine an adjustment valuefor modifying the parameter, wherein the adjustment value is determinedaccording to a functional relationship based on the operational valueand a factor, a desired driver behavior being maintained by thefunctional relationship; and an output operatively coupled with theprocessor and with an input of the safety event system, the outputselectively receiving the adjustment value for modifying the safetyevent system setting, and delivering the adjustment value to the safetyevent system for effecting a modification of the setting of the safetyevent system of the associated vehicle.
 16. The safety system as setforth in claim 15, wherein: the safety event system is danger detectionsystem.
 17. The safety system as set forth in claim 16, wherein: thesafety event system is a lane departure warning system.
 18. The safetysystem as set forth in claim 15, wherein: if the determined driver roadattention compliance indicates the associated driver is in aninattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time basedon the monitored condition.
 19. The safety system as set forth in claim18, wherein: the safety event system is a lane departure warning system;the monitored condition of the lane departure warning system is theassociated vehicle crossing a lane; if the determined driver roadattention compliance indicates the associated driver is in aninattention state, the operational value of the parameter of themonitored condition of the safety event system is modified by theadjustment value to warn the associated driver at an earlier time if theassociated vehicle crosses the lane.
 20. The safety system as set forthin claim 15, wherein: the safety event system is headway keeping aid.21. A method of modifying a setting of a safety event system, the methodcomprising: capturing an image of an associated driver disposed in anassociated vehicle and of an interior of the associated vehicle;generating driver head image data representative of the captured imageof the associated driver disposed in the associated vehicle and of theinterior of the associated vehicle; determining a facing direction ofthe head of the associated driver based on the driver head image data;generating driver head facing direction data representing the determinedfacing direction of the head of the associated driver; based on thedriver head facing direction data, determining an operational value ofthe driver road attention parameter of a monitored driver attentioncondition of the associated vehicle; comparing the determinedoperational value of the driver road attention parameter of themonitored driver attention condition of the associated vehicle and atleast one of i) recursively measured statistical values based on asufficient number of measurements derived at a sufficient speed of theassociated vehicle and ii) a statistical value derived from a histogramassociated with a straight ahead driver's head pose direction derived atthe sufficient speed of the associated vehicle; determining driver roadattention compliance in accordance with a result of the comparisonbetween a recommended value and the determined operational value of thedriver road attention parameter of the monitored driver attentioncondition of the associated vehicle; relating the determined driver roadattention compliance to an operational value of a parameter of amonitored condition of the safety event system; determining anadjustment value for modifying the parameter, wherein the adjustmentvalue is determined according to a functional relationship based on theoperational value and the functional relationship, a desired driverbehavior being maintained by the functional relationship; andtransmitting the adjustment value for modifying the setting of thesafety event system.
 22. The method of modifying a setting of a safetyevent system as set forth in claim 21, further including: modifying thesetting of the safety event system.
 23. The method of modifying asetting of a safety event system as set forth in claim 21, furtherincluding: determining the monitored driver attention condition based onan elapsed time the driver head facing direction data indicates theassociated driver last looked at a roadway along which the associatedvehicle is traveling.
 24. The method of modifying a setting of a safetyevent system as set forth in claim 21, further including: determining alocation of the driver's head relative to the one or more structures ofthe associated vehicle; and generating driver's head location data, thedriver's head location data representative of the determined location ofthe head of the associated driver relative to the one or more structuresof the associated vehicle.
 25. The method of modifying a setting of asafety event system as set forth in claim 24, further including:determining the operational value of the driver road attention parameterbased on the driver head facing direction data and the driver's headlocation data.
 26. The method of modifying a setting of a safety eventsystem as set forth in claim 25, further including: if the determineddriver road attention compliance indicates the associated driver is inan inattention state, modifying the operational value of the parameterof the monitored condition of the safety event system by the adjustmentvalue to warn the associated driver at an earlier time based on themonitored condition.
 27. The method of modifying a setting of a safetyevent system as set forth in claim 26, wherein the safety event systemis a lane departure warning system and the monitored condition is theassociated vehicle crossing a lane, the method further including: if thedetermined driver road attention compliance indicates the associateddriver is in an inattention state, modifying the monitored condition ofthe lane departure warning system by the adjustment value to warn theassociated driver at an earlier time if the associated vehicle iscrossing the lane.
 28. The method of modifying a setting of a safetyevent system as set forth in claim 26, wherein the safety event systemis a headway keeping aid and the monitored condition is a distance to aforward vehicle, the method further including: if the determined driverroad attention compliance indicates the associated driver is in aninattention state, modifying the monitored condition of the headwaykeeping aid by the adjustment value to warn the associated driver at anearlier time if the distance from the associated vehicle to the forwardvehicle is less than a predetermined headway distance.